fix(gatsby-source-contentful): reduce memory usage #37910
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
* fix(gatsby): validate sub plugins options of gatsby-plugin-mdx * revert hardcoded path to subplugins * validate subplugins if they are not under 'options.plugins' field --------- Co-authored-by: Michal Piechowiak <misiek.piechowiak@gmail.com>
* actual fix * other misc changes
- create-gatsby@3.9.0-next.1 - gatsby-cli@5.9.0-next.1 - gatsby-plugin-sharp@5.9.0-next.1 - gatsby-transformer-sqip@5.9.0-next.1 - gatsby@5.9.0-next.2
|
Great! Thank you so much for your work! |
This was referenced Jun 9, 2023
This was referenced May 13, 2024
This was referenced May 13, 2024
This was referenced May 14, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Along with #37782 and #37780 this PR allows a 4.9M node Contentful site to build with 24G of memory. Before these 3 PRs the site needed 64G memory and would OOM on data updates.
This PR does a few things:
During any iteration on nodes there is code to prevent the event loop from being blocked. This included adding a queue for calling createNode so that the queue can pause as needed to not block. This lowers memory and also generally speeds up the sourcing process.
While iterating through large lists, chunks of the list is set to undefined after being processed. This allows GC before the whole list is done being iterated on, keeping memory usage down.
On every data update every existing node was loaded into memory and iterated on and then GC'd at the end of sourceNodes. This was to search and build up backreferences. In this PR I changed it so it stores the minimal info about nodes needed for back references and keeps it in memory between data updates. You'd think this would increase memory usage but it substantially decreases is while also massively speeding up data updates. For the 4.9M node site a data update would OOM after 10 minutes. Now it succeeds after 45s to 3 min (depending on the update).