[torqueusers] What is the Maximum no of compute Nodes in torque

Gus Correa gus at ldeo.columbia.edu
Thu May 14 17:36:18 MDT 2009


Thanks Ken, Bogdan, Garrick, Chi

Very kind of you to answer so thoroughly to my questions,
which were mostly driven by curiosity, not necessity.

One day, budget permitting, we'll get beyond 1024 nodes,
and thanks to your insight and clarifications,
now I know I will still be running Torque!

Today, however, all clusters counted,
we have exactly 68 nodes ...
and no Torque scaling problems so far.  :)

Best,
Gus Correa
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------


Ken Nielson wrote:
> Gus,
> 
> Torque does not maintain open connections between the server and moms nor between the Mother Superior and sisters. The server and each mom have listen ports open where connections are made but the protocol is a request/reply so a new connection is opened for each new command and closed when the proper replies have been received. Communication also takes place over UDP, particularly on ports 15001 at the server and 15003 on sister nodes. Because UDP is connectionless PBS_NET_MAX_CONNECTIONS has no affect. 
> 
> Ken Nielson
> Cluster Resources
> 
> ----- Original Message -----
> From: "Gus Correa" <gus at ldeo.columbia.edu>
> To: "Torque Users" <torqueusers at supercluster.org>
> Sent: Wednesday, May 13, 2009 8:30:30 AM GMT -07:00 US/Canada Mountain
> Subject: Re: [torqueusers] What is the Maximum no of compute Nodes in torque
> 
> Hi Chi, list
> 
> Thank you!
> Great information!
> 
> Yes, I read that thread while it was going on,
> but I was just too lazy to search for it on the Torque list archives.
> 
> Thank you also for the information relating PBS_NET_MAX_CONNECTIONS=1024
> to the maximum number of sockets, select(), etc.
> I guessed there was some system related limit there, but didn't know 
> which/where to find it.
> 
> In any case, according to answers on that thread, one can have more than
> 1024 pbs_moms running.
> However, it is unclear to me how this will
> work with the 1024 limit on open sockets/connections.
> 
> What happens if more than 1024 pbs_moms try to connect to the pbs_server?
> Do some of them wait on a queue, or are they simply dropped,
> and keep trying until the server has available connections?
> 
> Thank you,
> Gus Correa
> ---------------------------------------------------------------------
> Gustavo Correa
> Lamont-Doherty Earth Observatory - Columbia University
> Palisades, NY, 10964-8000 - USA
> ---------------------------------------------------------------------
> 
> Chi Chan wrote:
>> I was having the same question, and the discussion can be found in
>> this discussion thread:
>>
>> http://www.clusterresources.com/pipermail/torqueusers/2009-April/008978.html
>>
>> Also, Linux's implementation of select() has a limit of 1024 sockets,
>> so I think that's why the Torque code has PBS_NET_MAX_CONNECTIONS
>> defined to 1024. If a larger number of simultaneous connections is
>> needed, you will need to change the definition in Torque and also the
>> system header file. (Another batch system changed the FD_SETSIZE
>> definition to 8K, see:
>> http://gridengine.sunsource.net/issues/show_bug.cgi?id=1502 ).
>>
>> --Chi
>>
>>
>> On Mon, May 11, 2009 at 2:48 AM, Rajiv Rajaian <rajiv.care at gmail.com> wrote:
>>> Hi
>>> Im using torque-2.0.0p8.
>>> What is the maximum no of the compute nodes that can be configured with
>>> torque..
>>> Is there any configuration to limit the no of compute nodes that can be
>>> configured..
>>>
>>> Thanks in Advance
>>> Rajiv
>>> Project associate,
>>> CARE,
>>> Anna university
>>>
>>> _______________________________________________
>>> torqueusers mailing list
>>> torqueusers at supercluster.org
>>> http://www.supercluster.org/mailman/listinfo/torqueusers
>>>
>>>
>> _______________________________________________
>> torqueusers mailing list
>> torqueusers at supercluster.org
>> http://www.supercluster.org/mailman/listinfo/torqueusers
> 
> _______________________________________________
> torqueusers mailing list
> torqueusers at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torqueusers



More information about the torqueusers mailing list