Uber, the taxi company with no vehicles, has provided an interesting insight into their network infrastructure. Being a global company, they need to be accessible by those locations that are IPv6 only. This is especially true as mobile phone networks become native IPv6. With the need to support DevOps and microservices, they need capacity in the data center. In 2016, they started the process. they share their story in their engineering blog entry Adopting the Next-Gen Internet Protocol.
Why do they need to move from IPv4?
Generous IP allocation and RFC1918 exhaustion
Uber uses the RFC1918 address space for addressing their infrastructure. This ensures that they don’t have problems with connecting with some of their customers on the Internet due to address conflicts. They have used more than half of their 10.0.0.0/8 address space. This is allocated as a /24 to each rack, with each rack taking at most 48 physical servers, so this allows each server to host 5 virtual environments.
Working backwards we can see that Uber could be using up to 8.4M IP addresses for their server environment using up to 1.5 million servers. (The video below tells us that 10s of 1000s of servers, so there is a lot of space in that 8.4M allocations).
Overlapping IP addresses
Uber are now acquiring companies, so these sometime have IP address allocations that overlap with current Uber space. Renumbering or remapping IP addresses when this occurs can be challenging, and could prevent some of the integration between organisations until resolved.
What Uber needed to change
In order to make the transition, Uber needed to make sure that the network architecture supported IPv6. They also needed to ensure that software and tooling supported IPv6, including making sure that their developers understood the challenges. And the vendor support is key to making a smooth transition.
The network architecture is both the hardware, automation tools, and the network design. The Uber datacentres are built using a Clos network (or leaf and spine) switch architecture with Juniper switches. Tooling is used to automate processes, including zero-touch provisioning, smart monitoring and IP address management.
The IPv6 address space (a /32, since you asked), is used solely for the internal network. This is split across the clusters, pods, racks and servers so that each rack is allocated a /64 block. Clusters aggregate the space to contiguous blocks. These can then be advertised by the IS-IS and BGP routing processes used in the Uber environment.
The tooling they have supports the consistent allocation of IPv6 addresses to each component in the data center cluster, e.g. rack subnets, pod subnets, loopbacks, out-of-band subnets, and point-to-point subnets.
The blog entry has quite a detailed section on the need for correct software support of IPv6. They detail several areas where code may have challenges when switching to IPv6 from IPv4.
- Code with prefix lengths set to common IPv4 prefixes, e.g. /24, are no longer suitable for IPv6 prefix lengths.
- Code that is written to split “a.b.c.d” into a tuple of (“a”,”b”,”c”,”d”) will fail to catch IPv6 addresses because they are separated by a colon (“:”) instead of a period (“.”).
- Code that expects to split ‘host:port’, e.g. “a.b.c.d:8080” into (“a.b.c.d”,”80″) will fail when “[2002:a:b:c::ff]:8080” is passed to it. Similarly, code that expects to join (“a.b.c.d”,”80″) into “a.b.c.d:8080” will create invalid IPv6 addresses in the format “2002:a:b:c::ff:8080”.
- Code with regex that used to match an IPv4 address will yield an invalid result when receiving an IPv6 address.
- Databases with hard-coded column sizes that store IPv4 addresses as “varchar(16)” due to “a.b.c.d\0” will be unable to store an IPv6 address.
– Uber Engineering Blog
The key here appears to make the software use DNS rather than hardcoded addresses. The DNS can then have the IPv6 addresses added to achieve the migration from IPv4 to IPv6.
This appears to be a very small entry, in the blog. Having the commitment of your vendors to your final environment is key. This is not only the vendors providing the software support, but I also suspect the validation of designs, and professional services involvement.
Make sure that the you have partners you can trust in the deployment of your IPv6 environment, those who have done it before, with provable results.
A successful IPv6 deployment for Uber still has a long way to go. Enabling the infrastructure for IPv6 is one thing.. Taking advantage of it all is another, including the built-in packet encryption and comprehensive QoS.
As well as reading the blog, Uber also have a YouTube video from one of their technical conferences covering their deployment.