[torqueusers] Submitting very short jobs

Axel Kohlmeyer akohlmey at cmm.chem.upenn.edu
Mon Oct 29 09:36:41 MDT 2012

On Sun, Oct 28, 2012 at 5:09 AM, Jon Tegner <tegner at renget.se> wrote:
> Hi,
> we have for a long time used maui/torque, and it has worked really well
> for out applications (mpi-jobs with many cores and execution times of
> days/weeks).
> But, we have other applications which instead consists of
> hundreds/thousands of single core very short applications (a few seconds).

> Haven't tried yet, but I imagine that due overhead it would be
> inefficient to use maui/torque in the same way as we have done previously.

yes, it is *extremely* inefficient to submit and run jobs in torque/maui
that don't run for at least half an hour or so.

however, that is not a problem. since you are submitting a shell script,
there is nothing keeping you from submitting a shell script that combines
hundreds of these short jobs into a single submission.

> I'm sure I'm not the first one with this kind of problem, and am just
> wondering if there are any best practices regarding this kind of problem.

in many cases, it may be advantageous to write scripts that
automate the process or creating and submitting the job bundle script.

if needed, this can be even expanded into having jobs processed
by a single parallel job using a manager/worker setup and then
reading a file with the individual commands. we use such a
(simple, self-written) tool for cases where the individual
calculations can take a varying amount of time and thus we also
have load balancing across the nodes (and to run efficiently on
machines where no single processor jobs are allowed).


> Thanks!
> /jon
> _______________________________________________
> torqueusers mailing list
> torqueusers at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torqueusers

Dr. Axel Kohlmeyer    akohlmey at gmail.com

Institute for Computational Molecular Science
Temple University, Philadelphia PA, USA.

More information about the torqueusers mailing list