IPv6, served with a side of AWS shenanigans

I’m going to kick this off by stating that I think the move to IPv6 is interesting. As long as I can remember, people have talked about IPv6, but the move never happened. Well, it seems as though that time is finally here. Back in July, AWS‘s “Chief Evangelist”, Jeff Barr, post this article that really put the wheels in motion. I’m going to do like everyone else has done, and share my 2 cents on the matter. But first, let’s have some fun with IPv6.

I had never worked with it before a couple of weeks ago. I never really had a need. I mean… I knew it was hexadecimal, and that there are a lot more available addresses than IPv4, but that’s about where my knowledge on that ended. And even today, I don’t profess to be a subject matter expert. All I can do is talk about my experience, which is exactly what I’m doing here. I started messing with it on my home network, and was immediately surprised to see that not only was NAT not a component, but every single device I had suddenly had a public IPv6 address. Given, this is still behind my pfSense firewall, so it’s not like all my gear is suddenly getting hammered with pings from foreign servers, but it was definitely a surprise. Considering that my ISP (Verizon FIOS) only gives me one IPv4 address (with no option for static), I wondered how many of these IPv6 addresses they gave me. They provided a /56 block. This gives me, a residential customer, 4,722,366,482,869,645,213,696 addresses. I’m concerned that the number is too big to have any real weight in your mind, dear reader, so please, take a gander at how you would actually pronounce that number:

four quadrillion,
seven hundred twenty-two trillion,
three hundred sixty-six billion,
four hundred eighty-two million,
eight hundred sixty-nine thousand,
six hundred forty-five thousand,
two hundred thirteen,
six hundred ninety-six

Yep. That’s a big number. Not only does it dwarf the roughly 4 billion public IPv4 addresses, but it obviously raised this question: how many IPv6 addresses are there?! Hold onto your hats, kids; this one’s a doozie. There are 340,282,366,920,938,463,463,374,607,431,768,211,456 IPv6 addresses. Even counting the number of commas is a bit daunting at this point, so allow me to spell this one out for you as well:

three hundred forty undecillion,
two hundred eighty-two decillion,
three hundred sixty-six nonillion,
nine hundred twenty octillion,
nine hundred thirty-eight septillion,
four hundred sixty-three sextillion,
four hundred sixty-three quintillion,
three hundred seventy-four quadrillion,
six hundred seven trillion,
four hundred thirty-one billion,
seven hundred sixty-eight million,
two hundred eleven thousand,
four hundred fifty-six

Unless you were playing Cookie Clicker at some point, you probably never even stumbled into that number in your life. Undecillion…. That’s a whole lot of -illions. And four quadrillion of them are mine! I admit, I did make a comment to my friend, the Keyboard Sloth, about what might happen if we one day run out of IPv6 addresses. He made the valid point that we will likely move away from IP-based networking far before we run out of addresses. As for me… I’m assuming it wouldn’t start to become an issue until we become an interplanetary species (which would be pretty neat, I think). Once we have four or five planets worth of people, maybe we’ll need to start looking for a new way of doing things. Anyway, this is a very complicated and excited transition to see the world go through right now, and I really look forward to seeing what interesting network products and software come out of this. Speaking of products and software, let’s review what AWS has been planning for this, then you can be as annoyed as I am.

Jeff Barr, in the aforementioned article, drops this little nugget of advice:

This change reflects our own costs and is also intended to encourage you to be a bit more frugal with your use of public IPv4 addresses and to think about accelerating your adoption of IPv6 as a modernization and conservation measure.

I, unlike most people, am in a very unique position at work where I actually CAN accelerate our adoption of IPv6. So I did! I designed a new architecture for our apps that would live in a IPv6-only VPC. The plan was to maintain two public subnets where load balancers would live, and two private subnets where our app servers/containers would live. A basic internet gateway (IGW) for the load balancers, and then an egress-only IGW for the private subnets so the containers can make requests out. Cloudflare has a trusty switch in the “network” settings of your domain that allow you to accept IPv4 AND IPv6 requests regardless of what protocol the origin server requires. Also, worth noting here is that Cloudflare does not charge for this. Anyway, Cloudflare can send the requests to the load balancers via the IGWs, and most of our outbound requests were only going to other AWS resources, so this seemed like a pretty easy way to do things.

DENIED.

Let’s talk about all the main couple of reasons this didn’t work.

1. I quickly learned that load balancers in AWS are either IPv4 only, or “dual stack”. In order to support this, your subnet cannot be “IPv6 native”. This isn’t a huge deal though; it’s only local IP addresses, so we’re not shelling out anything for them.
2. When I tried to provision an ECS cluster via Pulumi, I got hit with an error that it was unable to pull the image from ECR. Remember my egress-only IGW? It wasn’t going to work. ECR doesn’t accept IPv6 requests.

As previously stated, the first issue isn’t the end of the world. I set the subnet up to support IPv4, and I set the load balancer to “dual stack”, and then it was fine. The thing I really take issue with here is that ECR won’t communicate on IPv6. This means that I can’t just use an egress-only IGW. Because I’m now forced to use IPv4, I need to either implement NAT gateways for each private subnet, or I need to set up VPC endpoints that allow communication between the private subnets and ECR without. Additionally, for the sake of transparency, I did find out that one of the external API’s we use also doesn’t support IPv6, so the NAT Gateways were needed regardless. The egress cost of using NAT Gateways to connect to ECR will be cheaper than setting up the VPC endpoints (at least for now), so that‘s the route we’re going. I’m still peeved about this, though.

No, this cost isn’t going to break the bank for anyone configuring their architecture this way. It’s not going to stop anyone from proceeding with an architecture like this either. However, my issue is that the AWS article implies that there is a cost benefit to moving to IPv6, and when I made the attempt, I was still met with a need for IPv4 when access their own services. I wouldn’t have a single complaint if the only issue was the third party services we are using, but we have AWS telling us to move to IPv6, then we have AWS not accommodating that migration while still pushing to charge us for IPv4 addresses.

Maybe I’m just expecting more from AWS than I should, but it feels disingenuous to me. AWS did not announce any kind of caveats or conditions regarding their ability to support IPv6; they’re just leaving it to people like me to find out on our own if we can actually go without IPv4. And the answer is “no, we cannot survive strictly on IPv6”. Considering most of the world isn’t ready for it, that’s not surprising, but it is a big point of contention for me that the people telling us to move to IPv6 do not support it end-to-end, but are still going to move forward with increasing our costs anyway. I will continue to work on minimizing my IPv4 footprint in AWS to try to mitigate these costs.