[torquedev] IP version-agnostic Address Representation

David Beer dbeer at adaptivecomputing.com
Thu Aug 25 17:06:10 MDT 2011

----- Original Message -----
> David Beer wrote:
> ----- Original Message -----
> Ken Nielson wrote:
> ----- Original Message -----
> From: "Donald Neal" <dmneal at wand.net.nz> To: "Torque Developers
> mailing list" <torquedev at supercluster.org> Sent: Thursday, August
> 11, 2011 6:05:59 PM
> Subject: [torquedev] IP version-agnostic Address Representation
> There are a range of cases in Torque where an IP address is
> represented
> by a 32-bit object. This is something of a problem where the IP
> address
> may actually be 128 bits long.
> I see two distinct cases here. In one the 32-bit object is being
> used
> as
> a key (as with AvlNode). In this case I propose initially to take
> the
> lower-order 32 bits of an IPv6 address and keep going as now. This
> is
> simple and effiicient, but would definitely give rise to
> collisions
> which will confuse people in future.
> So the key does in future need to change. My inclination is to use
> a
> struct created for the purpose containing two uint64_t's. But
> there
> are
> definitely other ways of doing this.
> In the other case, the address is not used as a key (as in struct
> pbsnode, say). That makes it viable to use either a struct
> containing
> the minimum fields necessary or a sockaddr*. I'm inclined towards
> the
> latter on the grounds that keeping down the number of struct types
> in
> use makes life simpler, But again there are clearly other ways of
> doing
> this.
> Does anyone have a view on this? Does anyone have any other reason
> to
> use a package like gmp, which I don't see as needed for this
> purpose
> alone?
> - Donald Neal The current scheme TORQUE uses to identify trusted
> hosts is to use
> the host name and IPv4 ip address. The 32-bit IP address is used
> as a key to store trusted addresses in an AVL tree. The convenient
> thing about using this scheme is that connections are verified
> without the need to read any data from the stream. With the
> 128-bit IPv6 address the AVL tree is broken. We could change the
> AVL-tree to take a 128 bit key and then pad the IPv4 addresses. (I
> am just brain-storming).
> Another thought is to have the server assign a random number to
> each MOM and then distribute those numbers across the cluster.
> This would require a change in the protocol. It also requires a
> reading of the incoming stream to get the number.
> Just some thoughts.
> Ken From the (limited) discussion, I'm inclined to conclude
> i) Noone has another use for gmp, so general-purpose large objects
> are
> not the answer.
> ii)) Replacing the 32-bit key in AvlNode with a struct of two
> uint64_t's
> is as good an idea as anything else.
> So life looks to be simpler if all AvlNodes have a 128-bit key in
> place
> of a 32-bit one. This becomes more complex if the same node
> identifier
> is used both with an AvlNode and with a resizable_array.
> Is there a simple explanation anywhere of what the resizable_array
> struct is intended to be used for? This is a question about design
> philosophy rather than existing code. The resizable_array struct is
> being used in many places to replace linked lists. This struct
> actually has the capability of maintaining a fixed order, so in a
> way it is still a linked list that resides in an array.
> The struct is used (in 4.0) to store jobs, nodes, queues, and tasks.
> As far as my intention for its use, I wanted to use it as an easy
> way for storing objects in arrays where the number of objects could
> change. I have tried to make it into a flexible and widely usable
> struct. Is there anything more specific you'd like to know about it?
> David Well, a general-purpose representation of a node's address
> can't be smaller than the address, which may be 128 bits. As it
> stands resizable_array is indexed by int. So the options look to me
> to be
> i) Replace the existing resizable_array with one indexed by an object
> which can hold up to 128 bits.
> ii) Create a second kind of resizable_array which is indexed by a
> 128-bit object, leaving the existing resizable_array unchanged.
> iii) Maintain a structure holding a 1:1 mapping between a
> general-purpose network address representation and the value used to
> index resizable_arrays.
> I don't like any of these much. I suspect option ii may result in
> less complexity of thinking if not less code than the others, but I
> welcome more informed opinions.

resizable_array holds void pointers, so you can store any struct you wish in there (a struct of two int64s, for example). The index is just an index, it doesn't have to correspond to the object. If you wanted to write things so as to store the ip addresses in a resizable array, I suggest that you look at the way jobs are stored:

job storage uses a resizable array and a hash table (the hash table just quickly looks up the index for where the job is stored). Doing things this way, you could store the ip addresses in a resizable array and still have quick lookups. For that matter, maybe there is a way to use the hash table for this as well.



More information about the torquedev mailing list