Monorepo and Nx support

[ascorbic]

Currently we can only handle monorepos if the Next site root directory (that contains next.config.js) can be set as base in the netlify.toml or UI. This means that it needs a package.json that installs all dependencies. It either needs to be self-contained, or it needs to be in a yarn workspace. This is because of the way Netlify handles monorepos.

The workaround of prepending cd the/site/root before the build command does not currently work for Next sites, because build plugins always run from the base directory, meaning the plugin can't find all the Next files. This also doesn't work for monorepos that need commands to be run from the root. This is a particular problem for Nx sites, as they have no way of running commands from the app directory.

When building a Next site, the build plugin needs to know two things:

1. The directory that contains the next.config.js. Currently the plugin assumes this is cwd(): i.e. the base. This is why the cd workaround fails.
2. The distDir (i.e. the .next directory). In most cases we can find this out from the config at 1. The exception is with Nx, which I'll cover more below.

Information that can be set by the user:

1. The base dir, which is currently where we assume next.config.js is, and where we run the build command.
2. The publish dir. By default this is out, and is where we move the files from distDir.
3. The distDir, which is set in next.config.js. By default this is .next, except with Nx, which rewrites it on the fly unless it has been changed by the user to something else, in which case it's left alone.

Possible solutions:

1. If the user sets the publish dir, we might assume that the next.config.js is in its parent directory. From this we can find the distDir.
   • Pros: Doesn't require any config. Works for most setups.
   • Cons: if the publish dir isn't a sibling of the Next config then this fails. It also fails if the distDir can't be found from the next.config.js. Both of the are the case by default with Nx, which changes distDir on the fly to a directory outside of the site root.
2. Allow the user to manually set a Next root directory.
   • Pros: Avoids the issue with next config not being a sibling of the publish dir.
   • Cons: doesn't solve the Nx issue, as the distDir can't be found from the config.
3. Allow the user to manually set a Next root and a distDir in the plugin config.
   • Pros: Works for all monorepos.
   • Cons: Needs two new config options. Can be confusing.
4. ✅ Solution 1, but require that Nx users set the distDir to a specific value, which overrides the behaviour where it rewrites it to a parent directory that we can't work out.
   • Pros: Doesn't require any new config options. We can check the config to ensure that the user has set the value.
   • Cons: Needs special handling for Nx. Forces user to change their site config.
5. Solution 1, plus special case handling for Nx: try reading details of the workspace from the config.
   • Pros: Doesn't need any special work by the user, or any config options.
   • Cons: More work. May not be possible: needs more investigation of Nx APIs. May be fragile if Nx changes APIs.

Monorepo types

The following project types have been tested

• All node_modules in Next subdirectory
• All node_modules at top level, Next in subdirectory
• Nx
• Lerna with yarn workspace
• Lerna with npm
• yarn workspace (?)
• TypeScript project reference (?)

[lindsaylevine]

this writeup is 🔥 . thanks for putting it together! :)

thoughts on the possible solutions:

1. sounds like the current implementation. obviously not functional outside of sites where the site is based from the project root and next.config.js is in the project root.
2. now that i think about it, this was actually pizzafox's approach (you may have noticed this when adding status stuff to the caching logic). https://github.com/jonahsnider/netlify-cache-nextjs/blob/master/src/index.ts#L22
3. if distDir as a plugin input is only needed for monorepo setups like nx (and not for the majority of users), i think this should be ok. i think, no matter what, our monorepo support will have to be well-documented because i do still think people's approaches can be very complicated i.e. the nx outputPath option can be very confusing paired with all of these other dist-related/output-related settings.
4. necessary evil for nx to spare all the other use cases extra complexity :'( +1 for documenting well
5. 😅 definitely feels like a nice-to-have, but it does seem like we have a nontrivial number of nx users. this also feels related to con 2 of solution 2.

if we go the route of nextRoot, i also think maybe nextSiteRoot or siteRoot might be worth thinking about as names instead of nextRoot. maybe its just me but the word next in there alone just has me thinking it could mean multiple things. if we're just writing nextRoot into the src though (without it as an input option), then doesn't matter imo.

[pzi]

Great write-up indeed, thanks for that.

If I may, I will add another weird scenario onto the pile of oddities:

I have replaced lerna with yarn workspaces (simpler for my use case) and got the following set up:

netlify.toml

[build]
  base = "/web/"
  publish = "/web/"
  command = "yarn build"

[[plugins]]
  package = "@netlify/plugin-nextjs"

next.config.js

const nextConfig = {
  reactStrictMode: true,
  target: 'serverless',
};

Netlify UI Deploy config

Base directory: /web
Build command: yarn build
Publish directory: Not set

Build log

7:25:14 PM: Different build dir detected, going to use the one specified in the Netlify configuration file: 'web' versus '/web' in the Netlify UI
7:25:14 PM: Different publish path detected, going to use the one specified in the Netlify configuration file: 'web/web' versus '' in the Netlify UI

...

7:26:16 PM: ────────────────────────────────────────────────────────────────
7:26:16 PM:   Netlify Build                                                 
7:26:16 PM: ────────────────────────────────────────────────────────────────
7:26:16 PM: ​
7:26:16 PM: ❯ Version
7:26:16 PM:   @netlify/build 12.6.0
7:26:16 PM: ​
7:26:16 PM: ❯ Flags
7:26:16 PM:   deployId: 60d31a0f43c7f9149616ca17
7:26:16 PM: ​
7:26:16 PM: ❯ Current directory
7:26:16 PM:   /opt/build/repo/web
7:26:16 PM: ​
7:26:16 PM: ❯ Config file
7:26:16 PM:   /opt/build/repo/web/netlify.toml
7:26:16 PM: ​
7:26:16 PM: ❯ Context
7:26:16 PM:   production
7:26:17 PM: ​
7:26:17 PM: ❯ Loading plugins
7:26:17 PM:    - @netlify/plugin-nextjs@3.4.2 from netlify.toml

...

7:27:05 PM: ────────────────────────────────────────────────────────────────
7:27:05 PM:   6. Deploy site                                                
7:27:05 PM: ────────────────────────────────────────────────────────────────
7:27:05 PM: Starting to deploy site from 'web/web'

Note the reference to a folder called `'web/web' which my project does not have....

And the pipe was successful 🤷🏼

Edit: note, if this is the wrong place to post the above, happy to move if you let me know how/where you want it :)

[ascorbic]

The issue is that "publish" is relative to "base". You should usually leave it alone, and it will default to "out" for Next.js sites

[pzi]

it will default to "out"

I am using next build not next export though?

[ascorbic]

Yes, that's where the build plugin should be putting it

[ascorbic]

I have a draft PR that implements option 4. It needs testing with Nx sites particularly, though I'd like tests from other monorepo types too eventually. Instructions on trying it out here: #434

[lindsaylevine]

@pzi i believe /web/web is because your base is /web, and the plugin will add a dir (if it doesnt exist) in your "project/site root" for the publish dir.

i'm not sure though, are you saying the build/deploy works when you say "the pipe was successful"?

[pzi]

i'm not sure though, are you saying the build/deploy works when you say "the pipe was successful"?

Yes, sorry. The build & deploy worked 😬 which felt odd because of the web/web folder situation

[pzi]

I will try and use the above WIP @ascorbic I am just mindful that I am on a free plan and I am burning through my build minutes. All these trial & error commits, builds sucked up 1/3 of my allocation 😅

I finally got a stable build & deploy again, albeit using yarn workspaces instead of lerna. However, I wonder if there is another variable that may need to be considered when dealing with mono repos:

I had a bunch of "nothing" deploys because I don't have a netlify.toml file in the root folder of my repo. I got this:

repo/
  package.json
  web/
    package.json
    netlify.toml
  studio/
    package.json
    netlify.toml

So, if I don't set any deploy config via UI, it results in "nothing" deploys.

If I change the UI base dir to /web and /studio respectively, CI picks up the right toml files and "additionally" uses them, so everything is then based on base dir of the UI plus the base in the toml file resulting in web/web and the next plugin picks up the right dist dir. Hope that makes any sense whatsover 😬

PS: Probably should mention that I have set up 2 sites (1 for studio and 1 for web)
PPS: I realise I may not be telling you anything new here so I will step away from this for a little bit and take a break.

[ascorbic]

Fixed in #434

[KevTale]

Hi @ascorbic ,

I'm not sure to fully understand how I'm supposed to setup my nx monorepo for deploying nextjs apps with netlify.

Here is my file structure:

org/
  apps/
    foo/
      next.config.js
      netlify.toml
    bar/
      next.config.js
      netlify.toml
  libs/
    ...
  package.json

On netlify UI, I've setup foo site to base = apps/foo

netlify.toml (for each app) is

 [build]
  command = "npm run build"
  publish = "out"

[dev]
  command = "npm run start"
  targetPort = 4200

 [[plugins]]
  package = "@netlify/plugin-nextjs"

next.config.js (for each app) is

// eslint-disable-next-line @typescript-eslint/no-var-requires
const withNx = require('@nrwl/next/plugins/with-nx');

/**
* @type {import('@nrwl/next/plugins/with-nx').WithNxOptions}
**/
const nextConfig = {
 nx: {
   // Set this to true if you would like to to use SVGR
   // See: https://github.com/gregberge/svgr
   svgr: false,
 },
 distDir: '.dist',
 target: 'serverless',
};

module.exports = withNx(nextConfig);

When deploying, i've got the follow error:

4:44:53 PM:   Error: This site does not seem to be using Next.js. Please run "npm install next" in the repository.
4:44:53 PM:   If you are using a monorepo, please see the docs on configuring your site: https://ntl.fyi/next-monorepos

What am I missing here?

[ascorbic]

Hi @KevTale
You need to set the base as the root of the repo, and if you need multiple sites per repo then don't set the publish dir in the toml, but rather in the UI. So set the publish dir as described in the Nx doc, but use the UI rather than the toml. The build plugin uses the publish dir to determine the which app is being built. It can't use the usual methods because of the way Nx rewrites the config on the fly at build time, and has the output outside of the app directory.