BTW, also It turned out that all checkpointing business is broken once you start to use requirements like "-l nodes=2:ppn=2 " when submit jobs. And pbs_server has to be patched in couple of places. Anton.