-
Notifications
You must be signed in to change notification settings - Fork 266
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add new stack parameter for enabling dualstack docker [PLT-2325] #1306
Merged
+37
−7
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yob
changed the title
Add new stack parameter for enabling dualstack docker
Add new stack parameter for enabling dualstack docker [PLT-2325]
Mar 28, 2024
Adds new `DockerNetworkingProtocol` Cloud Formation Parameter, for opting in to starting containers with both ipv4 and ipv6 addresses. This might be useful when running the elastic stack on dual stack and ipv6-only subnets. Prior to this change, elastic stack instances could run on dual stack and ipv6-only VPC subnets and many things continue to work as normal. However, docker builds and containers do not - containers are started with private ipv4 addresses and no ipv6 addresses. Attempts to connect to global or VPC ipv6 addresses from inside a container fail. When this new parameter is set to `dualstack`, containers will be assigned an ipv6 address and depending on the configuration of the VPC it's running in, containers may be able to connect to the outside world over ipv6. -- dualstack subnets When the new parameter is set to `dualstack` and the instances are running on a dualstack VPC subnet, we expect: * containers are started with a private ipv4 * containers are started with an ipv6 from 2001:db8::/32 * connecting to external resources over ipv4 works, with docker performing NAT from the container private ipv4 to the EC2 instance AWS assigned ipv4 * connecting to external resources over ipv6 works, with docker performing NAT from the container private ipv6 to the EC2 instance AWS assigned ipv6. NATing ipv6 feels weird, but it's the way docker 25.x works -- ipv6-only subnets When the new parameter is set to dualstack and the instances are running on a ipv6-only VPC subnet, here's what should happen: * containers are started with a private ipv4 * containers are started with an ipv6 from 2001:db8::/32 * connecting to external resources over ipv4 will fail with an error * connecting to external resources over ipv6 works, with docker performing NAT from the container private ipv6 to the EC2 instance AWS assigned ipv6. NATing ipv6 feels weird, but it's the way docker 25.x works * It's very likely your ipv6 subnet will have DNS64 enabled. In this case, DNS hostnames that return only A addresses will have a fake AAAA record added to the response. This record points to an AWS NAT gateway. The container should use the fake ipv6 address and the AWS NAT gateway will translate to ipv4 (for a fee). -- why 2001:db8::/32? https://chameth.com/ipv6-docker-routing/ is a good explaination of the issue. Given docker is NATing the containers ipv6 traffic, the natural ipv6 range to use would be fd00::/8 - the ULA range (https://en.wikipedia.org/wiki/Unique_local_address). That range will partially work, however on a dualstack subnet where containers have working ipv4 and ipv6 addresss, using fd00::/8 will make most requests default to ipv4 instead of ipv6. That's particularly undesirable in a CI environment on AWS because ipv4 traffic almost certainly goes via a NAT gateway with per Gb fees, and CI often generates a lot of AWS ingress traffic. By using 2001:db8::/32 the dualstack containers should default to using ipv6 in most cases where they have the choice of either protocol. This may avoid significant NAT gateway fees. 2001:db8::/32 is technically reserved for documentation use only. The cost benefits are significant though, and docker is using them privately so there should be no negative impact. -- why opt-in? It would be nice to just auto-detect when the instances have a valid ipv6 addresses assigned and automatically start docker in dualstack mode. However, there is significant potential to change the behavior of networking in the container and might have surprising side effects. This prosposes starting opt-in so we can start to build some experience with dualstack docker and AWS. Eventually, it would be great to make it Just Work.
I've rebased this on top of v6.20.0 |
DrJosh9000
reviewed
May 13, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
Co-authored-by: Josh Deprez <jd@buildkite.com>
DrJosh9000
approved these changes
Jun 12, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🥦
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds new
DockerNetworkingProtocol
Cloud Formation Parameter, for opting in to starting containers with both ipv4 and ipv6 addresses. This might be useful when running the elastic stack on dual stack and ipv6-only subnets.Prior to this change, elastic stack instances could run on dual stack and ipv6-only VPC subnets and many things continue to work as normal. However, docker builds and containers did not - containers are started with private ipv4 addresses and no ipv6 addresses. Attempts to connect to global or VPC ipv6 addresses from inside a container fail. When this new parameter is set to
dualstack
, containers will be assigned an ipv6 address and depending on the configuration of the VPC it's running in, containers may be able to connect to the outside world over ipv6.dualstack subnets
When the new parameter is set to
dualstack
and the instances are running on a dualstack VPC subnet, we expect:ipv6-only subnets
When the new parameter is set to dualstack and the instances are running on a dualstack VPC subnet, here's what should happen:
why 2001:db8::/32?
https://chameth.com/ipv6-docker-routing/ is a good explaination of the issue.
Given docker is NATing the containers ipv6 traffic, the natural ipv6 range to use would be fd00::/8 - the ULA range
(https://en.wikipedia.org/wiki/Unique_local_address). That range will partially work, however on a dualstack subnet where containers have working ipv4 and ipv6 addresss, using fd00::/8 will make most requests default to ipv4 instead of ipv6. That's particularly undesirable in a CI environment on AWS because ipv4 traffic almost certainly goes via a NAT gateway with per Gb fees, and CI often generates a lot of AWS ingress traffic.
By using 2001:db8::/32 the dualstack containers should default to using ipv6 in most cases where they have the choice of either protocol. This may avoid significant NAT gateway fees.
2001:db8::/32 is technically reserved for documentation use only. The cost benefits are significant though, and docker is using them privately so there should be no negative impact.
why opt-in?
It would be nice to just auto-detect when the instances have a valid ipv6 addresses assigned and automatically start docker in dualstack mode. However, there is significant potential to change the behavior of networking in the container and might have surprising side effects. This prosposes starting opt-in so we can start to build some experience with dualstack docker and AWS. Eventually, it would be great to make it Just Work.
trying this out
I've been testing this on VPCs created by terraform, using the common public vpc module.
Dual stack VPCs are configured like this:
IPv6-only Subnets are configured like this:
In both cases I pass the VPC and subnet IDs into the stacks as parameters, to avoid teh cloudformation stack creating its own VPC.: