Dropbox deploys IPv6 at the edge

Dropbox‘s blog has a post from Haowei Yuan and colleagues, about engineering the Dropbox network to support IPv6 at the edge. Dropbox is serving about 15% of their requests via IPv6 each day. With IPv4 addresses no longer available, they need to support IPv6 to support the growth of their service in areas which are IPv6 only.

The Point of Presence

The Dropbox edge network consists of several globally distributed points of presence (PoPs), These include Layer 4 load-balancers and Layer 7 proxy servers. The Layer 4 load-balancing occurs using the IPVS kernel module, and proxy servers using Nginx. The virtual IP for a system is announced on the Internet via BGP, as well as advertised via DNS.

Traffic is delivered to the virtual IP of the load-balancers. These then forward the request to one of a bank of proxies in an IPv6 in IPv6 tunnel. The proxies are dual-stack, running both IPv4 and IPv6. They talk IPv4 internally to a similar set of equipment in the data centres.

The proxies use Direct Server Response to send the data back to the originating address directly. DSR off-loads the outbound traffic from the IPVS platform, allowing a little more scalability than if they had to support bidirectional traffic. In the current environment, IPv6 stops at the PoPs and becomes IPv4 internally.

Building an IPv6 network

Dropbox uses IS-IS as their internal routing protocol, and use a single topology to provide consistency between IPv4 and IPv6 routing. The IPv6 IS-IS runs over the IPv6 enabled links added into the network since late 2016. Having converted the IS-IS infrastructure, they added BGP with separate sessions for IPv4 and IPv6. Routing policies, however, are consistent across both protocols.

On top of the BGP environment, they also use MPLS-TE to link the PoPs and data centres together. The MPLS-TE mesh uses the same set of LSP tags for both IPv4 and IPv6.

In less than six months, Dropbox had completed an IPv6 roll-out across their network infrastructure; covering all links, points of presence and data centres.

Each PoP announces an IPv6 /48 network using BGP, ensuring that traffic through a PoP is kept local. Each IPv6 VIP /128 address is advertised as an IPv6 /64 to save memory. It also encodes the IPv4 address as the lowest four octets; allowing easy operational understanding of both the IPv4 internal numbering and the external IPv6.

Looking at the application stack

The primary change that IPv6 required was to ensure that the Nginx proxies and data centre systems could handle the IPv6 address. This was added in the X-Forwarded For: header entry. They also had to adjust some of the TCP parameters in order to deal the with tunnelled infrastructure. Tunnelling the IPv6 traffic inside IPv4 with the larger IPv6 headers meant a reduction of the TCP MSS to 1400 bytes.

Also, they faced the usual issues with IPv6 transitions, including applications not recognising IPv6 addresses, and regular expressions only supporting IPv4. Their GeoIP library needed to work with IPv6. And finally, access control lists required updating to work with IPv6. The most prominent issue they faced was the with ICMPv6, used for neighbour discovery. RFC6192 has some guidelines for protecting the router control plane that helps in these situations.

Some analysis of Dropbox’s IPv6 traffic

The top 10 countries by volume of IPv6 traffic overall are Belgium (38%), Luxembourg (30%), Greece (26%), United States (26%), Germany (25%), Estonia (23%), Ireland (21%), Switzerland (21%), Brazil (21%), and India (17%). There are also some comparisons of performance for TCP over both IPv6 and IPv4. IPv6 seems to provide better performance from the numbers provided. Dropbox uses the BBR TCP congestion-control mechanism developed by Google, as a lot of their traffic is bulk transfer.