[torqueusers] pbsmake -- a way to specify jobs with dependencies

Tim Spriggs imoverclocked at gmail.com
Wed Jul 11 22:36:15 MDT 2012


Hi All,

  I've been working with torque/maui and my user base for a while and
I've found the ability to submit jobs with dependencies is pretty
useful. However, it's also pretty messy if you have any kind of
less-than-trivial dependency graph and you are trying to capture job
IDs via a shell. Given some other push/pull factors I decided to
develop a utility that takes a make-like syntax where each "make"
target gets turned into a job and submitted to the job scheduler. Here
is a quick example:

---

> cat test
#!/usr/bin/pbsmake -f

123: testing
	@Output_Path localhost:~/job_outputs/
	echo 123
	date

testing:
	@Output_Path localhost:~/job_outputs/
	echo testing
	date
	sleep 2

> ./test
testing(41672.torqueserver...) scheduled
123(41673.torqueserver...) scheduled

> sleep 4
> cat ~/job_outputs/testing.o41672 ~/job_outputs/123.o41673
testing
Wed Jul 11 20:58:57 MST 2012
123
Wed Jul 11 20:59:00 MST 2012

---

  Of course, other parameters can be specified per-target such as
queue, resource limits, mail settings, etc.

  pbsmake is written against the python pbs module and uses a single
connection to send all jobs to the scheduler which makes it fairly
efficient for sending many jobs at once.

  pbsmake supports wildcards with wildcard dependencies. This allows
all kinds of mischievous stuff to be written (exercise best left to
the user) and a quick example of a wildcard target:

---
#!/usr/bin/pbsmake -f

a-%: %
	@Output_Path localhost:~/job_outputs/
	echo a ${pm_target_match}

b-%: %
	@Output_Path localhost:~/job_outputs/
	echo b ${pm_target_match}

c-%: %
	@Output_Path localhost:~/job_outputs/
	echo c ${pm_target_match}

d:
	@Output_Path localhost:~/job_outputs/
	echo d

> ./test-wild a-b-c-d
d(41682.torqueserver...) scheduled
c-d(41683.torqueserver...) scheduled
b-c-d(41684.torqueserver...) scheduled
a-b-c-d(41685.torqueserver...) scheduled

> cat ~/job_outputs/*.o4168[2345]
a b-c-d
b c-d
c d
d
---

  Finally, to make debugging targets with dependencies and wildcards
much easier, there is a graphviz output mode that will output a
digraph that can be piped to dotty (or other graphviz tools) to view
the dependencies before they are actually sent out. Yes, it will
visualize circular dependencies appropriately:

---
> ./test-wild -d a-b-c-d
digraph pbsmakefile {

t_0 -> t_1;

t_2 -> t_0;

t_3 -> t_2;

t_2 [label="b-c-d"];
t_3 [label="a-b-c-d"];
t_0 [label="c-d"];
t_1 [label="d"];
}
---

  The above command can be invoked as "./test-wild -d a-b-c-d | dotty
-" to popup an appropriate graph.

  For those of you who are still reading, congratulations! You win the
URL to the released versions of pbsmake as well as the development
repository for it:

http://pirlwww.lpl.arizona.edu/~tims/pbsmake/
https://github.com/imoverclocked/pbsmake

  I'd like to take a second to thank Chris Van Horne for all of the
development work he has done to get pbsmake where it is now while also
putting up with my sometimes much less than rock solid plans for how
pbsmake should behave in certain corner cases and turning them into
specs and code.

  Hopefully others may find this useful in their work with torque/pbs.

Cheers,
-Tim


More information about the torqueusers mailing list