[torqueusers] Limit NFS IO Speed?

Mike Dacre mike.dacre at gmail.com
Thu Nov 8 18:44:34 MST 2012


Hi Craig, 

Yeah, it seems that all the traffic is on 2049.  Right now the system is busy, so this is a bad time to test.  I will try to get basic traffic shaping working when the system is quieter and see if I can track down the problem.

Thanks for the help,

Mike 


On Thursday, November 8, 2012 at 1:41 PM, Craig Tierney wrote:

> On Thu, Nov 8, 2012 at 1:58 PM, Mike Dacre <mike.dacre at gmail.com (mailto:mike.dacre at gmail.com)> wrote:
> > Hi Craig,
> > 
> > Yes you are right, it is outbound only. What I have been trying to do is
> > limit the traffic from the client side. The settings below are from the
> > individual slave nodes. It is very difficult to quiet the system as it is
> > being actively used, however I have tried to apply the setting below with
> > very restrictive limits - around 1MB/s per node - without seeing any
> > discernible slowdown of traffic. I also tried from the server side just to
> > see if it would slow down reads (which are outbound from the server's
> > perspective), also without any effect.
> > 
> > Is there any chance the problem is that the NFS is async? I know that is a
> > problem for cgroups, but I didn't think it would be a problem here because
> > tc/iptables filtering/mangling is at the packet level.
> > 
> 
> 
> The traffic shaping isn't perfect. But I don't see why this wouldn't
> work. I have no idea if async would make a difference, but I would
> suspect no. Are you sure than NFS is only using port 2049?
> 
> Have you tried experiments where you tried some basic shaping between
> two nodes to make sure you have it working? I would find your
> favorite tutorial, try and get tc working between two nodes on a port
> for iperf. When you have that working then convert it to NFS.
> 
> Craig
> 
> > 
> > Thanks,
> > 
> > Mike
> > 
> > On Thursday, November 8, 2012 at 12:34 PM, Craig Tierney wrote:
> > 
> > Mike,
> > 
> > I can't follow all of your settings because I am not an expert in the
> > tc command, however I have played with it. What I remember is that
> > traffic shaping only is outbound. I would quiet the system and run
> > some tests where you do some writes from multiple nodes, then reads.
> > Do they seem throttled at all? Is the performance difference in each
> > direction?
> > 
> > Craig
> > 
> > On Thu, Nov 8, 2012 at 12:44 PM, Mike Dacre <mike.dacre at gmail.com (mailto:mike.dacre at gmail.com)> wrote:
> > 
> > Hi Everyone,
> > 
> > I am having an unusual problem: I have an Infiniband network connecting my
> > nodes and I have a RAID array mounted over NFS on every node. What is
> > happening is that the nodes are reading/writing too fast to the NFS mount,
> > and the IO of the array is being maxed out, which results in terrible
> > performance for interactive commands (e.g. ls). I have tried traffic
> > shaping with iptables and tc on both the server and slave nodes with no
> > success at all. I am not even certain those commands are working properly
> > on an IPoIB NIC (ib0).
> > 
> > The TC command I am trying is:
> > 
> > $TC qdisc add dev ib0 root handle 1:0 htb
> > $TC class add dev ib0 parent 1:0 classid 1:1 htb rate 50mbps ceil 50mbps
> > $TC class add dev ib0 parent 1:1 classid 1:2 htb rate 10mbps ceil 20mbps
> > $TC qdisc add dev ib0 parent 1:2 sfq
> > $TC filter add dev ib0 parent 1:0 protocol ip u32 match ip sport 2049
> > 0xffff flowid 1:2
> > $TC filter add dev ib0 parent 1:0 protocol ip u32 match ip dport 2049
> > 0xffff flowid 1:2
> > 
> > or
> > 
> > $TC qdisc add dev ib0 root handle 1:0 htb
> > $TC class add dev ib0 parent 1:0 classid 1:1 htb rate 50mbps ceil 50mbps
> > $TC class add dev ib0 parent 1:1 classid 1:2 htb rate 10mbps ceil 20mbps
> > $TC qdisc add dev ib0 parent 1:2 sfq
> > $TC filter add dev ib0 parent 1:0 protocol ip prio 1 handle 6 fw flowid
> > 1:2
> > 
> > With the following iptables:
> > 
> > /sbin/iptables -A POSTROUTING -t mangle -o ib0 -p tcp -m multiport --sport
> > 2049 -j MARK --set-mark 6
> > /sbin/iptables -A POSTROUTING -t mangle -o ib0 -p tcp -m multiport --sport
> > 2049 -j RETURN
> > /sbin/iptables -A POSTROUTING -t mangle -o ib0 -p udp -m multiport --sport
> > 2049 -j MARK --set-mark 6
> > /sbin/iptables -A POSTROUTING -t mangle -o ib0 -p udp -m multiport --sport
> > 2049 -j RETURN
> > /sbin/iptables -A POSTROUTING -t mangle -o ib0 -p tcp -m multiport --dport
> > 2049 -j MARK --set-mark 6
> > /sbin/iptables -A POSTROUTING -t mangle -o ib0 -p tcp -m multiport --dport
> > 2049 -j RETURN
> > /sbin/iptables -A POSTROUTING -t mangle -o ib0 -p udp -m multiport --dport
> > 2049 -j MARK --set-mark 6
> > /sbin/iptables -A POSTROUTING -t mangle -o ib0 -p udp -m multiport --dport
> > 2049 -j RETURN
> > 
> > I don't want to go back to ethernet NFS and only use Infiniband for MPI
> > because I want to maximize IO when there is a single node doing all the IO,
> > and ethernet is just too slow for that.
> > 
> > Any thoughts?
> > 
> > Thanks,
> > 
> > Mike
> > 
> > _______________________________________________
> > torqueusers mailing list
> > torqueusers at supercluster.org (mailto:torqueusers at supercluster.org)
> > http://www.supercluster.org/mailman/listinfo/torqueusers
> > 
> > _______________________________________________
> > torqueusers mailing list
> > torqueusers at supercluster.org (mailto:torqueusers at supercluster.org)
> > http://www.supercluster.org/mailman/listinfo/torqueusers
> > 
> > 
> > 
> > _______________________________________________
> > torqueusers mailing list
> > torqueusers at supercluster.org (mailto:torqueusers at supercluster.org)
> > http://www.supercluster.org/mailman/listinfo/torqueusers
> > 
> 
> _______________________________________________
> torqueusers mailing list
> torqueusers at supercluster.org (mailto:torqueusers at supercluster.org)
> http://www.supercluster.org/mailman/listinfo/torqueusers
> 
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.supercluster.org/pipermail/torqueusers/attachments/20121108/3aaa5861/attachment.html 


More information about the torqueusers mailing list