[torqueusers] What is the Maximum no of compute Nodes in torque

Gus Correa gus at ldeo.columbia.edu
Tue May 12 10:12:52 MDT 2009


Hi Rajiv, Ken, Jerry, list

The Torque documentation is quite terse about the hard limits.
The Admin Manual has some information about large clusters
in Appendix F, but no words about a possible hard limit on the
number of nodes:

http://www.clusterresources.com/torquedocs21/a.flargeclusters.shtml

Maui (if you use it) has an even more terse page about large clusters:

http://www.clusterresources.com/products/maui/docs/a.ilargeclusters.shtml

The closest thing seems to be this (in src/include/server_limits.h)

#define PBS_NET_MAX_CONNECTIONS 1024  /* increased from 256 */

This is probably the point that Ken Nielson referred to,
and suggests a maximum of 1024 connections to ... pbs_moms (i.e nodes).

Maybe I didn't understand Ken's explanation, but I still have some 
questions:

1) If Torque doesn't allow more than 1024 simultaneous connections to 
pbs_moms, can one run simultaneous jobs on more than 1024 nodes?
2) Or does this require increasing the PBS_NET_MAX_CONNECTIONS value?
3) Would a larger value require also changes in the OS limits,
to work properly?

Grep in the src/include directory may tell something,
but I already tried grep -i for "node", "pbs_node", "max",
etc, and no real gem came up.

I am glad that we have only small clusters here!  :)

Gus Correa
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------

Ken Nielson wrote:
> I have been looking at the code to see if there is a hard limit set to the maximum number of nodes you can have but I do not see any. There is a limit of 1024 simultaneous connections but Torque uses a request reply protocol so each command sequence starts with a connect followed by an exchange of data and a closing of the socket. This may cause a bottleneck for large clusters but it in no way inhibits the number of nodes possible.
> 
> Ken Nielson
> Cluster Resources
> 
> ----- Original Message -----
> From: "Jerry Smith" <jdsmit at sandia.gov>
> To: "Rajiv Rajaian" <rajiv.care at gmail.com>
> Cc: torqueusers at supercluster.org
> Sent: Tuesday, May 12, 2009 8:01:11 AM GMT -07:00 US/Canada Mountain
> Subject: Re: [torqueusers] What is the Maximum no of compute Nodes in torque
> 
> Rajiv,
> 
> I am not sure of the defined Maximum #, but we currently have it scaled 
> to ~4500 nodes (pbs_mom) to a single pbs_server on one of our clusters.
> 
> Jerry
> 
> 
> Rajiv Rajaian wrote:
>> Hi
>> Im using torque-2.0.0p8.
>> What is the maximum no of the compute nodes that can be configured 
>> with torque..
>> Is there any configuration to limit the no of compute nodes that can 
>> be configured..
>>
>> Thanks in Advance
>> Rajiv
>> Project associate,
>> CARE,
>> Anna university
> 
> _______________________________________________
> torqueusers mailing list
> torqueusers at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torqueusers
> _______________________________________________
> torqueusers mailing list
> torqueusers at supercluster.org
> http://www.supercluster.org/mailman/listinfo/torqueusers



More information about the torqueusers mailing list