Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[NEXT-1143] Dev mode slow compilation #48748

Open
1 task done
jeengbe opened this issue Apr 23, 2023 · 396 comments
Open
1 task done

[NEXT-1143] Dev mode slow compilation #48748

jeengbe opened this issue Apr 23, 2023 · 396 comments
Assignees
Labels
linear: next Confirmed issue that is tracked by the Next.js team.

Comments

@jeengbe
Copy link

jeengbe commented Apr 23, 2023

⚠️ this original post has been edited by @timneutkens to reflect this comment ⚠️

Changes in the past week

I've been investigating this over the past week. Made a bunch of changes, some make a small impact, some make a large impact. Here's a list:

You can try them using npm install next@canary.

Help Investigate

In order to help me investigate this I'll ideally need an application that can be run, if you can't provide that (I understand if you can't) please provide the .next/trace file.

If possible follow these steps which would give me the best picture to investigate:

  • npm install next@canary (use the package manager you're using) -- We want to make sure you're using the very latest version of Next.js which includes the fixes mentioned earlier.
  • rm -rf .next
  • start development using the NEXT_CPU_PROF=1 and NEXT_TURBOPACK_TRACING=1 (regardless of if you're using Turbopack, it only affects when you do use Turbopack) environment variable. E.g.:
    • npm: NEXT_TURBOPACK_TRACING=1 NEXT_CPU_PROF=1 npm run dev
    • yarn: NEXT_TURBOPACK_TRACING=1 NEXT_CPU_PROF=1 yarn dev
    • pnpm: NEXT_TURBOPACK_TRACING=1 NEXT_CPU_PROF=1 pnpm dev
  • Wait a few seconds
  • Open a page that you're working on
  • Wait till it's fully loaded
  • Wait a few seconds
  • Make an edit to a file that holds a component that is on the page
  • Wait for the edit to apply
  • Wait a few seconds
  • Make another edit to the same file
  • Wait a few seconds
  • Exit the dev command (ctrl+c)
  • Upload the CPU traces put in the root of the application directory to https://gist.github.com
  • Upload the .next/trace file to https://gist.github.com -- Please don't run trace-to-tree yourself, as I use some other tools (e.g. Jaeger) that require the actual trace file.
  • If you're using Turbopack upload the .next/trace.log as well, if it's too large for GitHub gists you can upload it to Google Drive or Dropbox and share it through that.
  • Upload next.config.js (if you have one) to https://gist.github.com
  • Share it here

Known application-side slowdowns

To collect things I've seen before that cause slow compilation as this is often the root cause:

  • If you're on Windows, disable Windows Defender, it's a known cause of extreme slowdowns in filesystem access as it sends each file to an external endpoint before allowing to read/write
  • Filesystem slowness overall is what we've seen as the cause of problems, e.g. with Docker
  • react-icons, material icons, etc. Most of these libraries publish barrel files with a lot of re-exports. E.g. material-ui/icons ships 5500 module re-exports, which causes all of them to be compiled. You have to add modularizeImports to reduce it, here's an example: long compile times locally - along with "JavaScript heap out of memory" since upgrade to NextJS 13 #45529 (comment)
  • Custom postcss config, e.g. tailwindcss with a content setting that tries to read too many files (e.g. files not relevant for the application)

This and other slowdown reports are currently the top priority for our team. We'll continue optimizing Next.js with webpack where possible.
The Turbopack team is currently working on getting all Next.js integration tests passing when using Turbopack as we continue working towards stability of Turbopack.

Original post

Verify canary release

  • I verified that the issue exists in the latest Next.js canary release

Provide environment information

Operating System:
      Platform: linux
      Arch: x64
      Version: #1 SMP Fri Jan 27 02:56:13 UTC 2023
    Binaries:
      Node: 18.13.0
      npm: 8.19.3
      Yarn: 1.22.18
      pnpm: 7.30.5
    Relevant packages:
      next: 13.3.1
      eslint-config-next: 13.3.1
      react: 18.2.0
      react-dom: 18.2.0

Which area(s) of Next.js are affected? (leave empty if unsure)

No response

Link to the code that reproduces this issue

https://github.com/DigitalerSchulhof/digitaler-schulhof

To Reproduce

Note that I have been unable to replicate this issue in a demo repository.

Describe the Bug

The issue is that Next.js is generally slow in dev mode. Navigating to new pages takes several seconds:

[next] ready - started server on 0.0.0.0:3000, url: http://localhost:3000
[next] info  - Loaded env from /home/jeengbe/dsh/digitaler-schulhof/.env
[next] warn  - You have enabled experimental feature (appDir) in next.config.js.
[next] warn  - Experimental features are not covered by semver, and may cause unexpected or broken application behavior. Use at your own risk.
[next] info  - Thank you for testing `appDir` please leave your feedback at https://nextjs.link/app-feedback
[next] event - compiled client and server successfully in 1574 ms (267 modules)
[next] wait  - compiling...
[next] event - compiled client and server successfully in 219 ms (267 modules)
[next] wait  - compiling /(schulhof)/Schulhof/page (client and server)...
[next] event - compiled client and server successfully in 3.6s (1364 modules)
[next] wait  - compiling /(schulhof)/Schulhof/(login)/Anmeldung/page (client and server)...
[next] event - compiled client and server successfully in 1920 ms (1411 modules)
[next] wait  - compiling /api/schulhof/auth/login/route (client and server)...
[next] event - compiled client and server successfully in 625 ms (1473 modules)
[next] wait  - compiling /(schulhof)/Schulhof/Nutzerkonto/page (client and server)...
[next] event - compiled client and server successfully in 1062 ms (1482 modules)
[next] wait  - compiling /(schulhof)/Schulhof/Nutzerkonto/Profil/page (client and server)...
[next] event - compiled client and server successfully in 1476 ms (1546 modules)
[next] wait  - compiling /(schulhof)/Schulhof/Nutzerkonto/Profil/Einstellungen/page (client and server)...
[next] event - compiled client and server successfully in 2.1s (1559 modules)

The only somewhat reasonable time would be 600ms for the API route /api/schulhof/auth/login/route, even though that is still quite a lot slower than what it should be given its size.

It also doesn't look right to compile ~1500 modules for each page, as most of them should be cached. The pages are not very different.

Even an empty API route takes several hundreds of ms. The following example contains solely type exports:

[next] wait  - compiling /api/schulhof/administration/persons/persons/settings/route (client and server)...
[next] event - compiled successfully in 303 ms (107 modules)

I am not exactly sure how to read trace trees, but what stands out is that there are (over multiple runs) several entry next-app-loader that take 2+ seconds to complete:

│  │  ├─ entry next-app-loader?name=app/(schulhof)/Schulhof/page&page=/(schulhof)/Schulhof/page&appPaths=/(schulhof)/Schulhof/page&pagePath=private-next-app-dir/(schulhof)/Schulhof/page.tsx&appDir=/home/jeengbe/dsh/digitaler-schulhof/app&pageExtensions=tsx&pageExtensions=ts&pageExtensions=jsx&pageExtensions=js&rootDir=/home/jeengbe/dsh/digitaler-schulhof&isDev=true&tsconfigPath=tsconfig.json&assetPrefix=&nextConfigOutput=! 1.9s

Find both dev and build traces here: https://gist.github.com/jeengbe/46220a09846de6535c188e78fb6da03e

Note that I have modified trace-to-tree.mjs to include event times for all events.

It also seems unusual that none of the modules have child traces.

Expected Behavior

Initial load and navigating should be substantially faster.

Which browser are you using? (if relevant)

No response

How are you deploying your application? (if relevant)

No response

From SyncLinear.com | NEXT-1143

@jeengbe jeengbe added the bug Issue was opened via the bug report template. label Apr 23, 2023
@joacub
Copy link

joacub commented Apr 24, 2023

same here, and in docker env is even worse, seems like is processing same files over and over without caching them.

@jinojacob15
Copy link

Same for me also dev env ,navigating to different pages via link component is pretty slow

@denu5
Copy link

denu5 commented Apr 25, 2023

+1 its same here, hitting the page first time seems fine but routing via links gets stuck

@joacub
Copy link

joacub commented Apr 25, 2023

last canary version has a better cold build times improvements, still slow like 2-5 seconds (in docker env) waiting but much better

the version im talking about is 13.3.2-canary.6

@denu5
Copy link

denu5 commented May 2, 2023

Hey, @jeengbe there have been some patch updates (13.3.1 -> 13.3.4) did it improve for you?

@jeengbe
Copy link
Author

jeengbe commented May 2, 2023

Hi @denu5,

unfortunately, I can't report any real performance changes since I opened this issue.

You might want to check out the above linked issue in the TypeScript repo though - might be related.

@joacub
Copy link

joacub commented May 2, 2023

As @jeengbe mention there is no performance improvement, there is also a lot of I/O I don’t know why, one request gets pretty much like 1gb-2gb of io. And it is very slow.

@jeengbe
Copy link
Author

jeengbe commented May 2, 2023

As @jeengbe mention there is no performance improvement, there is also a lot of I/O I don’t know why, one request gets pretty much like 1gb-2gb of io. And it is very slow.

Unfortunately, I can't confirm this for my case

image

@joacub
Copy link

joacub commented May 2, 2023

As @jeengbe mention there is no performance improvement, there is also a lot of I/O I don’t know why, one request gets pretty much like 1gb-2gb of io. And it is very slow.

Unfortunately, I can't confirm this for my case

image

That’s pretty good, in my case there is a lot of I/O, maybe is because I’m using material-ui but I think is too much even though.

@jeengbe
Copy link
Author

jeengbe commented May 2, 2023

As @jeengbe mention there is no performance improvement, there is also a lot of I/O I don’t know why, one request gets pretty much like 1gb-2gb of io. And it is very slow.

Unfortunately, I can't confirm this for my case
image

That’s pretty good, in my case there is a lot of I/O, maybe is because I’m using material-ui but I think is too much even though.

Possibly, it would align with what your trace shows: #48407 (comment)

@langfordG
Copy link

langfordG commented May 4, 2023

I see that slow route changes in dev mode are showing a '[Fast Refresh] rebuilding' message in the browser console. Sometimes it performs a full page reload when changing routes even if no files have been edited.

@timneutkens timneutkens added the linear: next Confirmed issue that is tracked by the Next.js team. label May 10, 2023
@timneutkens timneutkens changed the title Dev mode very slow navigation (Slow entry next-app-loader spans?) [NEXT-1143] Dev mode very slow navigation (Slow entry next-app-loader spans?) May 10, 2023
@AsathalMannan
Copy link

its slowing down the development..!

@vajajak
Copy link

vajajak commented May 21, 2023

Having the same issue here, in the Docker environment it's come to a point where it's almost unusable, and sometimes I even have to do a hard reload, after waiting too long for navigation. This is the case both with component from next/navigation, as with the router.push (useRouter hook imported from next/navigation). We're using Next.js 13.4.2.

@joacub
Copy link

joacub commented May 22, 2023

Having the same issue here, in the Docker environment it's come to a point where it's almost unusable, and sometimes I even have to do a hard reload, after waiting too long for navigation. This is the case both with component from next/navigation, as with the router.push (useRouter hook imported from next/navigation). We're using Next.js 13.4.2.

same here, it is almost not usable in dokcer enviorements, but also outside docker is very slow, something is not working nice. this is painfully slow.

@JoshApp
Copy link

JoshApp commented May 27, 2023

Yeah same for me. I used to remote develop inside our k8s cluster but dev --turbo is super slow inside a container and causes my health check endpoint to sigkill it regularly.

The whole app router is super slow when containerized in Dev mode.

It works perfectly fine when I run both on my local machine and connect it via reverse proxy. This way it's faster than the old setup (which was not significantly faster before) and takes advantage of preloading pages via next/link. I see inconsistencies in caching too where it's a mix of instant navigation or long builds (around 3.5k modules for some things) around 2-10 sec.

Also there is this weird thing happening that a page compiles just fine and then later it grinds to a halt being stuck in waiting for compiling forever until the pod is crashed.

@Rykuno
Copy link

Rykuno commented May 27, 2023

I love next, but this is a complete show stopper. Sometimes it takes 10+ seconds outside of docker for me on a Mac M2 to navigate one page.

This is insane.

@joacub
Copy link

joacub commented May 27, 2023

I love next, but this is a complete show stopper. Sometimes it takes 10+ seconds outside of docker for me on a Mac M2 to navigate one page.

This is insane.

Yep even more I get sometimes 50 seconds in a simple page, that’s because is also building other things related to that in pralllel I guess.

not even outside docker, i just make a test to work outside docker and timing is exactly the same no difference…. Is getting slower and slower

@joacub
Copy link

joacub commented May 27, 2023

Btw webpack lazy building cold is faster than turbopack 🙂 by far

@Rykuno
Copy link

Rykuno commented May 27, 2023

Btw webpack lazy building cold is faster than turbopack 🙂 by far

Yes! I'm surprised this is not more prevalent as an issue atm; unless turbo will somehow fix all of this in 13.5 and they're waiting to address it.

What configs do you have for the faster webpack builds? I've tried quite a bit and can't lower my built time by much. I need a temporary fix for this ASAP :(

@oalexdoda
Copy link

A month later no updates on this? Makes development on appDir absolutely impossible. @timneutkens ?

Linked a bunch of related issues on this:
#50332

@JunkyDeLuxe
Copy link

I confirm that next app dir on dev mode and dynamic routing are very very slow on docker now

@timneutkens
Copy link
Member

timneutkens commented Jun 6, 2023

Changes in the past week

I've been investigating this over the past week. Made a bunch of changes, some make a small impact, some make a large impact. Here's a list:

You can try them using npm install next@canary.

Help Investigate

In order to help me investigate this I'll ideally need an application that can be run, if you can't provide that (I understand if you can't) please provide the .next/trace file.

If possible follow these steps which would give me the best picture to investigate:

  • npm install next@canary (use the package manager you're using) -- We want to make sure you're using the very latest version of Next.js which includes the fixes mentioned earlier.
  • rm -rf .next
  • start development using the NEXT_CPU_PROF=1 environment variable. E.g.:
    • npm: NEXT_CPU_PROF=1 npm run dev
    • yarn: NEXT_CPU_PROF=1 yarn dev
    • pnpm: NEXT_CPU_PROF=1 pnpm dev
  • Wait a few seconds
  • Open a page that you're working on
  • Wait till it's fully loaded
  • Wait a few seconds
  • Make an edit to a file that holds a component that is on the page
  • Wait for the edit to apply
  • Wait a few seconds
  • Make another edit to the same file
  • Wait a few seconds
  • Exit the dev command (ctrl+c)
  • Upload the CPU traces put in the root of the application directory to https://gist.github.com
  • Upload the .next/trace file to https://gist.github.com -- Please don't run trace-to-tree yourself, as I use some other tools (e.g. Jaeger) that require the actual trace file.
  • Share it here

Known application-side slowdowns

To collect things I've seen before that cause slow compilation as this is often the root cause:

  • If you're on Windows, disable Windows Defender, it's a known cause of extreme slowdowns in filesystem access as it sends each file to an external endpoint before allowing to read/write
  • Filesystem slowness overall is what we've seen as the cause of problems, e.g. with Docker
  • react-icons, material icons, etc. Most of these libraries publish barrel files with a lot of re-exports. E.g. material-ui/icons ships 5500 module re-exports, which causes all of them to be compiled. You have to add modularizeImports to reduce it, here's an example: long compile times locally - along with "JavaScript heap out of memory" since upgrade to NextJS 13 #45529 (comment)
  • Custom postcss config, e.g. tailwindcss with a content setting that tries to read too many files (e.g. files not relevant for the application)

This and other slowdown reports are currently the top priority for our team. We'll continue optimizing Next.js with webpack where possible.
The Turbopack team is currently working on getting all Next.js integration tests passing when using Turbopack as we continue working towards stability of Turbopack.

@timneutkens timneutkens changed the title [NEXT-1143] Dev mode very slow navigation (Slow entry next-app-loader spans?) [NEXT-1143] Dev mode slow navigation Jun 6, 2023
@timneutkens timneutkens changed the title [NEXT-1143] Dev mode slow navigation [NEXT-1143] Dev mode slow compilation Jun 6, 2023
@timneutkens
Copy link
Member

Changed the initial post in this issue to reflect my reply above in order to ensure people see it as the first thing when opening the issue. I'm going to close the duplicate issues reporting similar slowdowns in favor of this one.

I'll need help from you all to ensure this thread doesn't spiral in "It is slow" comments that are not actionable when e.g. without traces / reproduction / further information. Thank you 🙏

@timneutkens
Copy link
Member

For some reason my trace.log is 209 MB in size. I don't think it's supposed to be that large.

That's expected, it holds much more data than the .next/trace including memory usage information and all data on individual function calls (hence why it's very helpful without needing all your code).

@Chai1310b
Copy link

Hi,
Thank you for your tips.
I uploaded my .next/trace of a project that take more than 30s to build, then ~10s when I edit a file
https://gist.github.com/Chai1310b/9f73885917b752f8fed8e97107de8510

Thanks ;)

@timneutkens
Copy link
Member

@AlaaCherif in your case it's a few libraries, some can still be optimized more in Turbopack, but just so you're aware:

  • packages/rake/dist/index.js imports the majority of antd: antd/es/index.js
    • packages/rake/dist/index.js is used in src/components/common/Header.jsx
  • src/components/lexical/RichTextEditor.jsx is quite slow to compile, could you share the code related to it so that we can dig further?
  • apps/nextjs/src/features/agenda/AgendaCarousel.jsx overall takes quite some compile time, around 35%.

@Chai1310b When you edit the file in your case you're importing @ant-design/icons/es/icons/index.js which is all icons in ant design (a ton of them).

On the initial compile there is a weird amount of time spent on sealing the build, around 24 seconds, did you customize the Next.js configuration (i.e. added webpack plugins)?

@timneutkens
Copy link
Member

Turbopack for development is now available as release candidate in Next.js 14.2: https://nextjs.org/blog/next-14-2

@AlaaCherif
Copy link

AlaaCherif commented Apr 12, 2024

@timneutkens

  • Rake is our internal component library that used to be compiled using Babel thus importing all of antd var _antd = require("antd"); (this is from the dist folder of babel compilation). I migrated yesterday to Vite and have seen small to no performance improvements. It has about a dozen simple UI components (e.g. Button, Modal ...) some use styled-components and other are Wrappers of antd components.
  • RichtTextEditor.jsx is a wrapper over Lexical. I included its content in the same Gist.
  • AgendaCarousel.jsx is a big component and it includes a lot of our projects business but it also uses Rake, so I'm not sure if that's expected since 35% of 30 seconds is still a lot.
  • I'm using next@14.2.0 and also tried using the canary versions.

Thanks again for the help

Update: I uploaded a new trace.log to the drive after the changes done as well as another trace file in the Gist.

@jellohouse
Copy link

Turbopack for development is now available as release candidate in Next.js 14.2: https://nextjs.org/blog/next-14-2

This works great 'next dev --turbo' for package.json dev script 🚀

@millsoft
Copy link

using the latest nextjs version. We are moving from pages to app directory. Pages in the app directory are fast. But pages in the pages directory are extremely slow. It takes like 7-10 minutes to get a response. Any idea why? I already tried to use a node profiler but could not see what the bottleneck is. Many file operations and string concat calls. We are already using turbopack.

image

@timneutkens
Copy link
Member

@millsoft please see the initial post of this issue, it outlines exactly how you can provide information that is critical for us to do any type of investigation if you don't want to share the codebase: #48748 (comment).

@timneutkens
Copy link
Member

@birukbelay checking in, are you able to provide the traces, happy to take a look if you can share them. Thanks!

@MarcusHSmith
Copy link

I'm having similar issues:

Following this guide: #48748 (comment)

Trace: https://gist.github.com/MarcusHSmith/dba462d850fdb69217027ab481edd3b2

Next config: https://gist.github.com/MarcusHSmith/68ad4b992164d5f4c1e3aaeb53a6e807

NEXT_TURBOPACK_TRACING=1 NEXT_CPU_PROF=1 yarn dev
yarn run v1.22.19
$ next dev -p 3002
▲ Next.js 14.2.2
- Local: http://localhost:3002/

✓ Starting...
> [PWA] PWA support is disabled
> [PWA] PWA support is disabled
✓ Ready in 3.2s
○ Compiling /landing ...
✓ Compiled /landing in 277.7s (2034 modules)
✓ Compiled in 2.8s (2034 modules)
GET /landing 200 in 260895ms

Any ideas how to improve our speed?

@timneutkens
Copy link
Member

CleanShot 2024-04-26 at 10 32 22@2x

@MarcusHSmith seems you've added postcss customization and it's taking forever to run that.

@timneutkens
Copy link
Member

For Tailwind CSS users: I've been able to find a pretty large slowdown with Tailwind CSS in general when content is misconfigured. This is not Next.js specific and would cause problems in any application, but since a lot of the reports nowadays are around Tailwind taking a long time to process I thought it'd be worth sharing here.

Worth having a look at this thread: https://twitter.com/timneutkens/status/1783851267237781574.

@AhmedBaset
Copy link
Contributor

Hey @timneutkens

Is turbo-trace-viewer.vercel.app a Vercel-internal? I can't find how to upload a trace file

Thanks in advance

@lpkobamn
Copy link

I had the same problem with nextjs appdir, compilation in devmode took up to 5-6 seconds.
I renamed global.scss to global.css and used @import "@carbon/styles/css/styles.css"
The compilation speed became 500-600ms and all styles were preserved

@timneutkens
Copy link
Member

@AhmedBaset correct, it's an internal tool we've been building that we'll be sharing at a later point, it's not fully ready yet, but useful for us 🙂

@so1ro
Copy link

so1ro commented May 3, 2024

It helped me. Deeply appreciate. Thank you 🫡

For Tailwind CSS users: I've been able to find a pretty large slowdown with Tailwind CSS in general when content is misconfigured. This is not Next.js specific and would cause problems in any application, but since a lot of the reports nowadays are around Tailwind taking a long time to process I thought it'd be worth sharing here.

Worth having a look at this thread: https://twitter.com/timneutkens/status/1783851267237781574.

@HallexCosta
Copy link

HallexCosta commented May 3, 2024

For Tailwind CSS users: I've been able to find a pretty large slowdown with Tailwind CSS in general when content is misconfigured. This is not Next.js specific and would cause problems in any application, but since a lot of the reports nowadays are around Tailwind taking a long time to process I thought it'd be worth sharing here.

Worth having a look at this thread: https://twitter.com/timneutkens/status/1783851267237781574.

Incredible. This really solved my problem of slow development, build and deployment. My Next.js app is small, but building and compiling took about 20~40 minutes to complete.

I already used the src folder inside my UI package, but due to an oversight I didn't change the tailwind compilation rule 🥲 .

@timneutkens Thanks for your post 😸 🚀.

Before
before-fix-tailwind-rule-compilation

After
after-fix-tailwind-rule-compilation

@sgup
Copy link

sgup commented May 4, 2024

It would be cool if there was a next doctor command that could find easy fixes like this

@meds
Copy link

meds commented May 5, 2024

CPU Profile

Trace

@timneutkens
Copy link
Member

@yasincandev this issue is specifically for discussing slowdowns in compilation, not for bundle sizes, feel free to open a discussion if you need help. Keep in mind that sharing a screenshot with numbers and the mention of 3 packages is not enough for people to be able to help you get an answer, you'll have to share more, i.e. code.

@timneutkens
Copy link
Member

@meds you're using Turbopack, please follow the steps here: #48748, specifically adding NEXT_TURBOPACK_TRACING=1 and making sure you share both .next/trace and .next/trace.log, they're separate files

timneutkens added a commit that referenced this issue May 6, 2024
## What?

Implements support for running the Turbopack trace server, which is the
websocket server that powers https://turbo-trace-viewer.vercel.app/ when
using `NEXT_TURBOPACK_TRACING=1 NEXT_TURBOPACK_TRACE_SERVER=1`.

Currently you have to manually run the server through the Turbo
repository which in practice means that only people working on Turbopack
are able to run it.

With the bindings implemented anyone should be able to run the trace
server.

Note that the traces that come out of Turbopack are very low level,
they're meant for optimizing Turbopack like finding slowdowns / large
memory usage / optimizing performance.

However, it's useful for people that want to peek into why their
application is slow to compile. I.e. we've used
https://turbo-trace-viewer.vercel.app to investigate reports in #48748.

This PR adds support for `trace.log` by default, so if you add
`NEXT_TURBOPACK_TRACING=1 NEXT_TURBOPACK_TRACE_SERVER=1` it will
automatically select the `trace.log` for the current instance of
Next.js. You can only have one trace server running at the same time.

### `next internal` 

In order to support running the trace server standalone, which is useful
for investigating trace files other people have shared, I've added a new
subcommand `internal` that is not covered by semver / use at your own
risk. It's meant for internal tools that are useful to be bound to the
version of Next.js, the turbo-trace-server is a great example of that as
it has an internal binary format for storing data that needs to match
the trace.log file.

If you want to take a look at `.next/trace` instead the new `next
internal` subcommand can be used for that:

```sh
# Replace [path] with a path to a file.
next internal turbo-trace-server [path]
```

For example:

```sh
next internal turbo-trace-server ~/Downloads/trace
```

Currently the trace server does not support loading multiple files, just
hasn't been implemented yet. Once we can load two or more files we can
load both `.next/trace` and `trace.log` when
`NEXT_TURBOPACK_TRACE_SERVER=1` and support multiple paths passed to
`next internal turbo-trace-server`.


### Turbopack upgrade


PR includes a Turbopack upgrade:

* vercel/turbo#8073 <!-- OJ Kwon -
feat(webpack-loaders): support dummy span interface -->
* vercel/turbo#8083 <!-- OJ Kwon - fix(webpack):
print resource, project path when relative calc fails -->
* vercel/turbo#8094 <!-- Tim Neutkens -
Implement bindings for Turbopack trace server -->


<!-- Thanks for opening a PR! Your contribution is much appreciated.
To make sure your PR is handled as smoothly as possible we request that
you follow the checklist sections below.
Choose the right checklist for the change(s) that you're making:

## For Contributors

### Improving Documentation

- Run `pnpm prettier-fix` to fix formatting issues before opening the
PR.
- Read the Docs Contribution Guide to ensure your contribution follows
the docs guidelines:
https://nextjs.org/docs/community/contribution-guide

### Adding or Updating Examples

- The "examples guidelines" are followed from our contributing doc
https://github.com/vercel/next.js/blob/canary/contributing/examples/adding-examples.md
- Make sure the linting passes by running `pnpm build && pnpm lint`. See
https://github.com/vercel/next.js/blob/canary/contributing/repository/linting.md

### Fixing a bug

- Related issues linked using `fixes #number`
- Tests added. See:
https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs
- Errors have a helpful link attached, see
https://github.com/vercel/next.js/blob/canary/contributing.md

### Adding a feature

- Implements an existing feature request or RFC. Make sure the feature
request has been accepted for implementation before opening a PR. (A
discussion must be opened, see
https://github.com/vercel/next.js/discussions/new?category=ideas)
- Related issues/discussions are linked using `fixes #number`
- e2e tests added
(https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs)
- Documentation added
- Telemetry added. In case of a feature if it's used or not.
- Errors have a helpful link attached, see
https://github.com/vercel/next.js/blob/canary/contributing.md


## For Maintainers

- Minimal description (aim for explaining to someone not on the team to
understand the PR)
- When linking to a Slack thread, you might want to share details of the
conclusion
- Link both the Linear (Fixes NEXT-xxx) and the GitHub issues
- Add review comments if necessary to explain to the reviewer the logic
behind a change

### What?

### Why?

### How?

Closes NEXT-
Fixes #

-->


Closes NEXT-3328
@jaskome
Copy link

jaskome commented May 7, 2024

Hey @timneutkens ,
I have also problems with slow compilation, it would be nice if you can take a look on it:

trace: https://gist.github.com/jaskome/18f97ed79117b022a38558196ff0851d
next.config: https://gist.github.com/jaskome/333099fddd65a9d1f562d618d83fac19

Thanks in advance

@msrks
Copy link

msrks commented May 14, 2024

hi @timneutkens
same issue. slow to compile..

trace: https://gist.github.com/msrks/2dc9cfef6ec3f42db3e318b3cae31f81

thanks in advance

@timneutkens
Copy link
Member

Hey everyone, as a bunch of people have asked about the trace viewer I've worked on publishing the tooling we use so that you can have a look yourself using next@canary.

Just so that everyone is on the same page this tooling is intentionally very low level, it's the tools we use to investigate slowdowns.

Specifically you can run these files:

  • .next/trace Default tracing that instruments webpack in development and production build
  • .next/trace.log Only written when using NEXT_TURBOPACK_TRACING=1 and is specific to Turbopack

You can run it using this command on next@canary:

next internal turbo-trace-server /path/to/.next/trace.log

It will link to https://turbo-trace-viewer.vercel.app/ where you can view the trace. Few notes on navigation:

  • Double click a span to zoom in
  • Right click a span to get more information
  • Use W / A / S / D or arrow keys to move around
  • The view modes allow you to switch between aggregated values and seeing the exact time for each span.
    • Aggregated is the default
    • in order means in the order of which they occured
    • by value means left-heavy, ordering the slowest spans from left to right -- Useful for finding which files are slow to process and such.

Only one trace file can be viewed at a time currently.

The trace viewer is still a work in progress, but already quite useful to us.

@Ziothh
Copy link

Ziothh commented May 23, 2024

EDIT: great work on the community support!

Old comment is there a similar tool for the webpack compiler?

When I run the command I get this error:

$ pnpx next@canary internal turbo-trace-server

Packages: +29
+++++++++++++++++++++++++++++
Progress: resolved 55, reused 30, downloaded 0, added 29, done
 ⚠ Turbopack trace server started. View trace at https://turbo-trace-viewer.vercel.app/
/Users/zioth/Library/pnpm/store/v3/tmp/dlx-73346/node_modules/.pnpm/next@14.3.0-canary.78_react-dom@19.0.0-rc-f994737d14-20240522_react@19.0.0-rc-f994737d14-20240522/node_modules/next/dist/build/swc/index.js:911
                    (customBindings ?? bindings).startTurbopackTraceServer(traceFilePath);
                                                 ^

Error: Failed to convert JavaScript value `Undefined` into rust type `String`
    at Object.startTurbopackTraceServer (/Users/zioth/Library/pnpm/store/v3/tmp/dlx-73346/node_modules/.pnpm/next@14.3.0-canary.78_react-dom@19.0.0-rc-f994737d14-20240522_react@19.0.0-rc-f994737d14-20240522/node_modules/next/dist/build/swc/index.js:911:50)
    at Module.startTurboTraceServerCli (/Users/zioth/Library/pnpm/store/v3/tmp/dlx-73346/node_modules/.pnpm/next@14.3.0-canary.78_react-dom@19.0.0-rc-f994737d14-20240522_react@19.0.0-rc-f994737d14-20240522/node_modules/next/dist/cli/internal/turbo-trace-server.js:14:20) {
  code: 'StringExpected'
}

Node.js v18.19.0

I didn't expect this to work but it would be nice if webpack users also had a similar way of viewing traces.

Thanks in advance!

@tom-carbontrail
Copy link

tom-carbontrail commented May 23, 2024 via email

@tiriana
Copy link

tiriana commented May 24, 2024

@timneutkens here is a gist with all the files mentioned in this comment

Gist: https://gist.github.com/tiriana/0e2dd12da8473d33ad6fcaeff182fac0

First file change after npm run dev is always very fast. Following changes take 15sec+ to be visible in the page.

It's much better with --turbo for me (refresh times below 1-2 sec)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
linear: next Confirmed issue that is tracked by the Next.js team.
Projects
None yet
Development

No branches or pull requests