[torquedev] Torque Semantics

"Mgr. Šimon Tóth" SimonT at mail.muni.cz
Tue Jun 15 09:11:34 MDT 2010


>>>  > For example requests like -l mem=1G are ignored during
>>>  > the scheduling process, but enforced on the node.
>>>
>>> I can't help but think whether the effort on bringing
>>> pbs_sched up to date with these sorts of things would
>>> be better spent on bringing in and maintaining Maui as
>>> the default scheduler instead; especially as I believe
>>> it handles this already.
>>
>> This was not about improving pbs_sched and making it up to date with
>> Maui features (that's actually a very stupid idea). This was about
>> documenting the Torques behaviour (server + qrun). I think that the
>> previous threads have proven that this behaviour is not well documented
>> an in some cases even ambiguous.
>>
>> I haven't used Maui, but I'm pretty sure that if Maui would be the
>> Torques scheduler we would be using Slurm right now instead of Torque.
>> Major reason being that Maui is designed around the limitation that each
>> server can have only one scheduler.
>>
>> Plus having Maui as a default scheduler would eliminate any development
>> effort in Torque. Maui is using Torque server just as job storage.
> 
> There is plenty of development work in torque that does not depend on
> pbs_sched (better job arrays coming out in 2.5.0 for example).

Well yes, exactly. And I would love to see this kind of development
effort to be sustained.

> I'm pretty sure you are the only person in the world using TORQUE
> pbs_sched for a non-trivial single FIFO scheduler configuration. Would
> you please share more details of your configuration so other
> developers have an idea where you're coming from?

The real issue is that we pretty much need the current semantics of
TORQUE server to be maintained as they are.

> I think you have a lot of patches that should get merged back
> upstream, but it would be helpful if everyone knew where you're coming
> from

OK, I will try to explain what we are doing just now.

Here are some documents:
Large one (probably not very interesting):
http://www.cesnet.cz/doc/techzpravy/2009/inter-cluster-scheduling/

Some newer documents from ISGC 2010:
http://www2.twgrid.org/event/isgc2010/slides/ISGC2010_matyska.pdf
http://www2.twgrid.org/event/isgc2010/abstract/PeertopeerCooperativeSchedulingArchitectureforNationalGridInfrastructure.pdf

Our current grid:
http://meta.cesnet.cz/cms/opencms/en/resources/hardware.html
http://meta.cesnet.cz/cms/opencms/en/resources/network.html

The whole point is transition from a single server installation of
PBSPro to a distributed system of Torque installations. One server for
such large cluster is extremely unreliable and doesn't allow any further
growth.

The first step is a simple M:N scheme, where we have M schedulers and N
servers talking to each other. Just a very light patch was needed for
this to work.

We have a lot of additional functionality in Torque because our grid is
heavily using virtualization technologies.

Our solution is based on pbs_sched because it was easy to modify. The
scheduling algorithm isn't important as it will be replace by a
sophisticated CPS solver in near future.

We do not rely on pbs_sched functionality, but we do rely on Torque
server functionality and current understanding of nodespec (with some
tweaks like adding support for generic resources).

-- 
Mgr. Šimon Tóth

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3366 bytes
Desc: S/MIME Cryptographic Signature
Url : http://www.supercluster.org/pipermail/torquedev/attachments/20100615/4237a585/attachment-0001.bin 


More information about the torquedev mailing list