PKI - Fix order of chain building writes #17772
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cipherboy
requested review from
kitography,
sgmiller,
jasonodonnell,
stevendpclark and
a team
When troubleshooting a recent migration failure from 1.10->1.11, it was noted that some PKI mounts had bad chain construction despite having valid, chaining issuers. Due to the cluster's leadership trashing between nodes, the migration logic was re-executed several times, partially succeeding each time. While the legacy CA bundle migration logic was written with this in mind, one shortcoming in the chain building code lead us to truncate the ca_chain: by sorting the list of issuers after including non-written issuers (with random IDs), these issuers would occasionally be persisted prior to storage _prior_ to existing CAs with modified chains. The migration code carefully imported the active issuer prior to its parents. However, due to this bug, there was a chance that, if write to the pending parent succeeded but updating the active issuer didn't, the active issuer's ca_chain field would only contain the self-reference and not the parent's reference as well. Ultimately, a workaround of setting and subsequently unsetting a manual chain would force a chain regeneration. In this patch, we simply fix the write ordering: because we need to ensure a stable chain sorting, we leave the sort location in the same place, but delay writing the provided referenceCert to the last position. This is because the reference is meant to be the user-facing action: without transactional write capabilities, other chains may succeed, but if the last user-facing action fails, the user will hopefully retry the action. This will also correct migration, by ensuring the subsequent issuer import will be attempted again, triggering another chain build and only persisting this issuer when all other issuers have also been updated. Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
In the previous commit, we identified an issue that would occur on legacy issuer migration to the new storage format. This is easy enough to detect for any given mount (by an operator), but automating scanning and remediating all PKI mounts in large deployments might be difficult. Write a new storage migration version to regenerate all chains on upgrade, once. Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
cipherboy
force-pushed
the
cipherboy-fix-order-of-chain-building-writes
branch
from
e0acf81
to
6c0a7de
Compare
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
cipherboy
force-pushed
the
cipherboy-fix-order-of-chain-building-writes
branch
from
6c0a7de
to
b964e77
Compare
stevendpclark
approved these changes
Nov 2, 2022
kitography
approved these changes
Nov 3, 2022
| @@ -43,7 +43,7 @@ func (sc *storageContext) rebuildIssuersChains(referenceCert *issuerEntry /* opt | |||
| // To begin, we fetch all known issuers from disk. | |||
| issuers, err := sc.listIssuers() | |||
| if err != nil { | |||
| return fmt.Errorf("unable to list issuers to build chain: %v", err) | |||
| return fmt.Errorf("unable to list issuers to build chain: %w", err) | |||
| // Ensure the chains are correct on the intermediate. By using the | ||
| // issuerId saved above, this ensures we didn't change any issuerIds, | ||
| // we merely updated the existing issuers. | ||
| intIssuer, err := sc.fetchIssuerById(intIssuerId) |
There was a problem hiding this comment.
this is really well done, thank you!
|
Thank you both! Merging... |
This was referenced Nov 3, 2022
jayant07-yb
pushed a commit
to jayant07-yb/hashicorp-vault-integrations
that referenced
this pull request
Mar 15, 2023
* Ensure correct write ordering in rebuildIssuersChains When troubleshooting a recent migration failure from 1.10->1.11, it was noted that some PKI mounts had bad chain construction despite having valid, chaining issuers. Due to the cluster's leadership trashing between nodes, the migration logic was re-executed several times, partially succeeding each time. While the legacy CA bundle migration logic was written with this in mind, one shortcoming in the chain building code lead us to truncate the ca_chain: by sorting the list of issuers after including non-written issuers (with random IDs), these issuers would occasionally be persisted prior to storage _prior_ to existing CAs with modified chains. The migration code carefully imported the active issuer prior to its parents. However, due to this bug, there was a chance that, if write to the pending parent succeeded but updating the active issuer didn't, the active issuer's ca_chain field would only contain the self-reference and not the parent's reference as well. Ultimately, a workaround of setting and subsequently unsetting a manual chain would force a chain regeneration. In this patch, we simply fix the write ordering: because we need to ensure a stable chain sorting, we leave the sort location in the same place, but delay writing the provided referenceCert to the last position. This is because the reference is meant to be the user-facing action: without transactional write capabilities, other chains may succeed, but if the last user-facing action fails, the user will hopefully retry the action. This will also correct migration, by ensuring the subsequent issuer import will be attempted again, triggering another chain build and only persisting this issuer when all other issuers have also been updated. Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com> * Remigrate ca_chains to fix any missing issuers In the previous commit, we identified an issue that would occur on legacy issuer migration to the new storage format. This is easy enough to detect for any given mount (by an operator), but automating scanning and remediating all PKI mounts in large deployments might be difficult. Write a new storage migration version to regenerate all chains on upgrade, once. Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com> * Add changelog entry Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com> * Add issue to PKI considerations documentation Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com> * Correct %v -> %w in chain building errs Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com> Signed-off-by: Alexander Scheel <alex.scheel@hashicorp.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Probably best to defer to the commit messages here:
As noted above, patching the issuer to set
manual_chain=selffollowed bymanual_chain=will be an adequate workaround.