[torquedev] Data-Is-String format
knielson at adaptivecomputing.com
Thu Jun 10 10:24:18 MDT 2010
On 06/10/2010 09:26 AM, Garrick Staples wrote:
> On Jun 10, 2010, at 12:20 AM, ekasit.kijsipongse at nectec.or.th wrote:
>> Hi all,
>> I'm wondering what is the benefit of using the Data-Is-String format in the
>> communication between pbs_server and pbs_sched, i.e. for reducing message size
>> maintaining readability? I've noticed a large overhead from sending and parsing
>> job information to pbs_sched if the number of jobs> 10,000. Is there any other
>> such as using database?
> DIS pre-dates all of us, so I don't think we can give you a good justification.
> I've always thought that we should add a final compression step. The data is highly highly repeated and should compress wonderfully.
I think compression is a great idea. One of the changes I have made to
the 2.4 branch is to allow the out-going buffer for tcp requests to
expand dynamically. It was originally hard coded at 262144 bytes and
Scinet had a 24000 processor job that exceeded the limit. We are
anticipating clusters of over 100,000 nodes with multiple cores. We
definitely need to reduce the payload on communication.
More information about the torquedev