Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(gatsby, gatsby-source-contentful): add public action to disable stale node checks #37782

Merged
merged 44 commits into from Apr 5, 2023

Conversation

TylerBarnes
Copy link
Contributor

@TylerBarnes TylerBarnes commented Mar 24, 2023

Along with #37780 (+ more changes I'll pr soon) this allows a 4.9M node Contentful site, that used to need 64Gi memory, build with only 24Gi memory.

The previous code blocked the event loop and loaded all nodes into memory. Memory quickly grew linearly (similar to my comment in the linked PR). With these changes memory usage stays more flat, but also being able to opt out entirely makes stale node checks much faster. With contentful there are so many nodes where getNode needs to be called to get the parent node and check if it's stale and that's really slow/costly.

New api:

import { hasFeature } from "gatsby-plugin-utils"

exports.sourceNodes = ({ actions }) => {
  if (hasFeature(`stateful-source-nodes`)) {
    actions.enableStatefulSourceNodes()
  } else {
    // fallback to old behavior where all nodes are iterated on and touchNode is called.
  }
}

@gatsbot gatsbot bot added the status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer label Mar 24, 2023
@TylerBarnes TylerBarnes removed the status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer label Mar 24, 2023
@TylerBarnes TylerBarnes added the topic: data Relates to source-nodes, internal-data-bridge, and node creation label Mar 25, 2023
@TylerBarnes TylerBarnes requested a review from pieh April 4, 2023 16:08
@TylerBarnes TylerBarnes merged commit fd168e0 into master Apr 5, 2023
31 checks passed
@TylerBarnes TylerBarnes deleted the feat/disable-node-gc-by-type branch April 5, 2023 23:13
@TylerBarnes TylerBarnes added this to To cherry-pick in V4 Release hotfixes via automation Apr 5, 2023
@TylerBarnes TylerBarnes added this to To cherry-pick in V5 Release hotfixes via automation Apr 5, 2023
@LekoArts LekoArts removed this from To cherry-pick in V4 Release hotfixes Apr 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic: data Relates to source-nodes, internal-data-bridge, and node creation topic: source-contentful Related to Gatsby's integration with Contentful topic: source-plugins Relates to the Gatsby source plugins (e.g. -filesystem)
Projects
V5 Release hotfixes
To cherry-pick
Development

Successfully merging this pull request may close these issues.

None yet

6 participants