[torquedev] Data-Is-String format

Ken Nielson knielson at adaptivecomputing.com
Thu Jun 10 10:24:18 MDT 2010


On 06/10/2010 09:26 AM, Garrick Staples wrote:
> On Jun 10, 2010, at 12:20 AM, ekasit.kijsipongse at nectec.or.th wrote:
>
>    
>> Hi all,
>>    I'm wondering what is the benefit of using the Data-Is-String format in the
>> communication between pbs_server and pbs_sched, i.e. for reducing message size
>> while
>> maintaining readability?  I've noticed a large overhead from sending and parsing
>> the
>> job information to pbs_sched if the number of jobs>  10,000.  Is there any other
>> options
>> such as using database?
>>      
> DIS pre-dates all of us, so I don't think we can give you a good justification.
>
> I've always thought that we should add a final compression step. The data is highly highly repeated and should compress wonderfully.
>
>    
Garrick,

I think compression is a great idea. One of the changes I have made to 
the 2.4 branch is to allow the out-going buffer for tcp requests to 
expand dynamically. It was originally hard coded at 262144 bytes and 
Scinet had a 24000 processor job that exceeded the limit. We are 
anticipating clusters of over 100,000 nodes with multiple cores. We 
definitely need to reduce the payload on communication.

Ken


More information about the torquedev mailing list