Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Relocate localterraform.com aliasing to backend configurators #33432

Conversation

brandonc
Copy link
Contributor

@brandonc brandonc commented Jun 26, 2023

Previously, remote and cloud backends would automatically alias localterraform.com as the configured hostname during configuration. This turned out to be an issue with how backends could potentially be used within the builtin terraform_remote_state data source. Those data sources each configure the same service discovery with different targets for localterraform.com, and do so simultaneously, creating an occasional concurrent map read & write panic when multiple data sources are defined.

localterraform.com is obviously not useful for every backend configuration. Therefore, I relocated the alias configuration to the callers, so they may specify when to use it. The modified design adds a new method to backend.Enhanced to allow configurators to ask which aliases should be defined.

Fixes #33333

Target Release

1.5.2

BUG FIXES

  • Fixed a panic when reading from terraform_remote_state data sources when multiple resources are defined

@brandonc brandonc force-pushed the TF-7327-using-multiple-terraform-remote-state-data-sources-can-cause-race-conditions-when-the-backend-configuring-localterraform-com branch from 7a7bdcf to d0f1ead Compare June 26, 2023 17:51
@brandonc brandonc force-pushed the TF-7327-using-multiple-terraform-remote-state-data-sources-can-cause-race-conditions-when-the-backend-configuring-localterraform-com branch from d0f1ead to d98c815 Compare June 26, 2023 18:17
@brandonc brandonc force-pushed the TF-7327-using-multiple-terraform-remote-state-data-sources-can-cause-race-conditions-when-the-backend-configuring-localterraform-com branch from d493598 to ae42c13 Compare June 26, 2023 18:44
@brandonc brandonc added the 1.5-backport If you add this label to a PR before merging, backport-assistant will open a new PR once merged label Jun 26, 2023
@brandonc brandonc marked this pull request as ready for review June 27, 2023 15:59
Copy link
Member

@apparentlymart apparentlymart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like a reasonable approach to me.

I left an inline comment about slightly tweaking the separation of concerns between command.Meta and the backend so that the backend is responsible only for providing the information and not for acting on it, but I'm not sure if that's better enough to be worth additional complexity; if doing that makes things significantly more complicated then we can probably live with the slightly strange separation of concerns this current changeset implies.

genericHost, err := svchost.ForComparison(aliasHostname)
if err != nil {
return fmt.Errorf("unusable alias hostname: %w", err)
}

// This won't be an error because, by this time, the hostname has been parsed and
// service discovery requests made against it.
targetHost, _ := svchost.ForComparison(b.hostname)

b.services.Alias(genericHost, targetHost)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does seem overall reasonable but I worry a little about this implicit assumption that b.services here is always the same object as the service discovery object used by the module and provider installers in terraform init. I imagine that's true today but it seems relatively easy for future changes to break that assumption if e.g. we end up needing to give different parts of Terraform different views of the discovery information for some reason yet to be determined.

While I'm not sure how likely that actually is, what bothers me is that someone could potentially do it without realizing that they are breaking a non-obvious assumption. For everything else about the service discovery system we treat it as something that's set once inside package main and then not mutated again, and so it would be reasonable for a new contributor to not notice this special exception to that.

I'm honestly not really sure what to suggest instead, though. 😖 One idea that came to my mind was to make this backend-level function just ask the question "what hostname should localterraform.com be an alias for?" and have it return targetHost instead of directly write it into its own b.services, and then it could be the caller's responsibility to actually register it with the service discovery object. I think that would also make this function infallible (no error cases) because the only possible error here is that the given hostname is invalid, and under this new design this method would not actually need to know the alias hostname at all.

That is admittedly only a marginal improvement since it still assumes that everyone's sharing a single disco object, but I imagine the caller that would be doing it would be command.Meta and that's already a sort of "central controller" for everything else and so I have a hunch that it's more likely to be able to ensure that everyone's svchost.Disco gets this information, even if in future it ends up having to register this against more than one discovery object for some reason.

Copy link
Contributor Author

@brandonc brandonc Jul 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@apparentlymart I tried my best to minimize the services modification being done by the backend, but it did turn out to be a marginal improvement because command.Meta is still modifying the service disco it doesn't own. But at least it's not happening one level further down in the backend. There was just no nice way I could see getting this alias information to the main package since the backend initialization is brokered by backendInit.

@brandonc brandonc force-pushed the TF-7327-using-multiple-terraform-remote-state-data-sources-can-cause-race-conditions-when-the-backend-configuring-localterraform-com branch from ae42c13 to 38840c1 Compare July 20, 2023 21:15
@brandonc brandonc force-pushed the TF-7327-using-multiple-terraform-remote-state-data-sources-can-cause-race-conditions-when-the-backend-configuring-localterraform-com branch from 38840c1 to 26ea507 Compare July 20, 2023 21:29
Copy link
Member

@apparentlymart apparentlymart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working through the compromise we discussed in the thread above! Although it's not quite the ideal of having the disco object be immutable once constructed, I do think this handling in Meta is more likely to stay correct under future maintenance because we're accustomed to that object being the shared state for everything in package command, at least.

I left another little thing inline; I'm hoping it's a relatively easy change and if I'm wrong about that (if it would be very time-consuming to do) I don't think it's crucial.

internal/backend/backend.go Outdated Show resolved Hide resolved
@brandonc brandonc force-pushed the TF-7327-using-multiple-terraform-remote-state-data-sources-can-cause-race-conditions-when-the-backend-configuring-localterraform-com branch from 26ea507 to 1e823ac Compare July 21, 2023 15:39
Previously, remote and cloud backends would automatically alias localterraform.com as the configured hostname during configuration. This turned out to be an issue with how backends could potentially be used within the builtin terraform_remote_state data source. Those data sources each configure the same service discovery with different targets for localterraform.com, and do so simultaneously, creating an occasional concurrent map read & write panic when multiple data sources are defined.

localterraform.com is obviously not useful for every backend configuration. Therefore, I relocated the alias configuration to the callers, so they may specify when to use it. The modified design adds a new method to backend.Enhanced to allow configurators to ask which aliases should be defined.
@brandonc brandonc force-pushed the TF-7327-using-multiple-terraform-remote-state-data-sources-can-cause-race-conditions-when-the-backend-configuring-localterraform-com branch from 1e823ac to c1a7303 Compare July 21, 2023 15:55
@brandonc brandonc merged commit 5fa956e into main Jul 21, 2023
6 checks passed
@github-actions
Copy link

Reminder for the merging maintainer: if this is a user-visible change, please update the changelog on the appropriate release branch.

Copy link

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 13, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
1.5-backport If you add this label to a PR before merging, backport-assistant will open a new PR once merged
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Terraform fatal error: concurrent map read and map write
3 participants