module: resolve format for all situations with auto module detection on #53044

dygabo · 2024-05-18T12:42:38Z

triggered by #53015
solves: #53016

this should be a consistent fix to always resolve the module format correctly.
Enabling module detection by default made a few other tests need some adjustments because in this case they don't generate errors anymore. e.g. test-esm-cjs-exports.js instead of error becasue a .mjs imports a .js with ESM syntax it now successfully imports it and generates the warning that this should be fixed to avoid the performance penalty.

Kindly please review and let me know what you think (if changes are necessary).

make test && make lint => green

@nodejs/loaders

nodejs-github-bot · 2024-05-18T12:42:43Z

Review requested:

@nodejs/loaders

dygabo · 2024-05-18T12:46:18Z

if the solution is feasible and finds approval, the --experimental-detect-module could already be removed in this PR. wdyt?

RedYetiDev · 2024-05-18T13:05:37Z

If you set detect_module to true, do all the tests (except the expected failures) pass?

RedYetiDev · 2024-05-18T13:07:00Z

Also,
🎉 Thank you for tackling this :-)

lib/internal/modules/esm/get_format.js

RedYetiDev

+1

Tip

While my review shows my support, I am not a core collaborator, and this review has no power / place in the approval process

GeoffreyBooth

Great work! Thanks for getting to the bottom of this.

GeoffreyBooth · 2024-05-18T13:48:07Z

src/node_options.h

@@ -110,7 +110,7 @@ class EnvironmentOptions : public Options {
 public:
  bool abort_on_uncaught_exception = false;
  std::vector<std::string> conditions;
-  bool detect_module = false;
+  bool detect_module = true;


I think we want to fix the format issue in one PR and then unflag the feature in the follow-up.

I will split this into two PRs once the failing test with __filename usage is sorted out

test/es-module/test-esm-cjs-exports.js

GeoffreyBooth · 2024-05-18T13:51:22Z

lib/internal/modules/esm/get_format.js

@@ -92,6 +92,12 @@ let typelessPackageJsonFilesWarnedAbout;
 function getFileProtocolModuleFormat(url, context = { __proto__: null }, ignoreErrors) {
  const { source } = context;
  const ext = extname(url);
+  const deduceFormat = (fromSource, fromUrl) => {


We should declare this at the top level. I'd also maybe name it determineFormat or something more specific to explain the cases when it's used.

We could also name it what the flag is named, something like detectModuleFormat

dygabo · 2024-05-18T14:09:13Z

Great work! Thanks for getting to the bottom of this.

Will check your comments and split it into two PRs, will happen maybe beginning of next week.

RedYetiDev · 2024-05-18T14:11:56Z

Great work! Thanks for getting to the bottom of this.

Will check your comments and split it into two PRs, will happen maybe beginning of next week.

Lovely!

lib/internal/modules/esm/get_format.js

test/parallel/test-worker-track-unmanaged-fds.js

lib/internal/modules/esm/get_format.js

dygabo · 2024-05-20T16:43:33Z

split done, this fixes the --experimental-detect-module edge cases, no unflagging of the option.
The unflagging is prepared and I can submit a PR after this one lands

aduh95 · 2024-05-20T17:29:56Z

lib/internal/modules/esm/get_format.js

+    if (Buffer.isBuffer(realSource)) {
+      // `containsModuleSyntax` requires source to be passed in as string
+      realSource = realSource.toString();
+    }


If I remove the following lines, all tests still pass. I suggest we remove them (unless we can have a test, but maybe it should be its own PR).

Suggested change

if (Buffer.isBuffer(realSource)) {

// `containsModuleSyntax` requires source to be passed in as string

realSource = realSource.toString();

}

pushed a commit to fix that. This will probably be necessary but I agree, an own PR with relevant test is better

GeoffreyBooth · 2024-05-20T19:59:08Z

test/es-module/test-esm-named-exports-detect-module.js

@@ -0,0 +1,13 @@
+// Flags: --experimental-detect-module --import ./test/fixtures/es-module-loaders/builtin-named-exports.mjs


Rather than make a new file for this, can it please get moved into the file that has the other tests with this fixture?

good point, made that with last commit

GeoffreyBooth · 2024-05-20T20:02:20Z

lib/internal/modules/esm/get_format.js

+ */
+function detectModuleFormat(source, url) {
+  try {
+    let realSource = source ?? readFileSync(url, 'utf8');


An earlier version of containsModuleSyntax expected a file path, where it would load a file from disk and then parse it all in C++ land. Doing readFileSync here means that we cross from JS to C++ to get the file contents and send that large string back across the boundary, only to then send it right back again from JS to C++ for containsModuleSyntax to work with it.

A more performant approach is probably to update containsModuleSyntax to allow for an undefined input in the source parameter, and another optional parameter that can be a file path. Then in this situation where the JS side doesn’t already have the source, we can tell the C++ side to both read it from disk and analyze it, without multiple trips crossing the boundary. cc @anonrig

I reintroduced the reading from file in the native code implementation for the case where code is not passed from the managed part. Slightly modified the logic as proposed in this comment.

The transpile case is currently the only one for which resolve cannot determine the format correctly. For that we only have the needed information during the load phase, not during the resolve hook is running.

GeoffreyBooth · 2024-05-20T20:03:07Z

test/es-module/test-esm-detect-ambiguous.mjs

@@ -155,7 +155,7 @@ describe('--experimental-detect-module', { concurrency: !process.env.TEST_PARALL
      });
    }

-    it('should not hint wrong format in resolve hook', async () => {
+    it('should hint format correctly for extensionles modules resolve hook', async () => {


Suggested change

it('should hint format correctly for extensionles modules resolve hook', async () => {

it('should hint format correctly for the resolve hook for extensionless modules', async () => {

GeoffreyBooth · 2024-05-20T20:05:25Z

lib/internal/modules/esm/get_format.js

+ * @param {string} source
+ * @param {URL} url
+ */
+function detectModuleFormat(source, url) {


The format property returned from resolve is optional. The addition of this helper seems like it’s becoming required; if we don’t know the format, we’re going to determine it, rather than waiting for load to figure it out. This means potentially reading the source from disk twice, unless you preserve what you get from the first read, which detectModuleFormat here currently doesn’t. Is there another way to fix the bug where we don’t necessarily read the source in resolve if we weren’t already doing so?

Reading the source within resolve is also problematic because that should be happening in the load hook. If you read file sources here, any custom hooks the user has registered for load won’t get applied before detection is run on whatever source is read here. So a TypeScript file would get misidentified as CommonJS because it can’t parse as ESM (or really, because it can’t parse at all) even if after the hooks are applied it’s transpiled into runnable ESM.

While I was thinking for a solution I had a similar chain of thoughts. This change makes resolve format resolution more reliable for most of the cases but as you also mentioned, it still does not cover the case where the load operation would have source modifications (e.g. some ts transpile step). Maybe a better solution would be to:

try compile as CJS

if possible => detected CJS

if not possible, try to compile as ESM

if possible => detected ESM

otherwise resolve detects format null (or undefined)

This would at least cover all the current tests, it will have the additional performance penalty of reading the file but it will always resolve the format correctly or not resolve it at all if it depends on load operation chain.

I will set the PR to draft until we decide on next steps from here.

Is there another way to fix the bug where we don’t necessarily read the source in resolve if we weren’t already doing so?

not as far as I can think now. Because we have to detect the module type that was not defined by the module author and for that, we currently try to compile it. If we have no source, we cannot make any assumptions that are valid. And it is better imo to stay with null or undefined in this case instead of trying to announce commonjs knowing that this might be wrong and changed later by the load step. This cannot be reliably solved without looking at the code of the module. Or do you see some alternative to it?

Concerning the performance, the whole auto detection mechanism is anyway a performance penalty because we parse/compile the source for it now as well. And we generate the warning that the user should fix it by specifying the type here

aduh95 · 2024-05-27T13:48:00Z

The C++ linter is failing

test/fixtures/es-modules/require-esm-throws-with-loaders.js

test/es-module/test-esm-loader-hooks.mjs

Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>

nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. esm Issues and PRs related to the ECMAScript Modules implementation. needs-ci PRs that need a full CI run. labels May 18, 2024

RedYetiDev requested a review from GeoffreyBooth May 18, 2024 13:05

RedYetiDev added the loaders Issues and PRs related to ES module loaders label May 18, 2024

RedYetiDev reviewed May 18, 2024

View reviewed changes

lib/internal/modules/esm/get_format.js Outdated Show resolved Hide resolved

RedYetiDev approved these changes May 18, 2024

View reviewed changes

GeoffreyBooth reviewed May 18, 2024

View reviewed changes

aduh95 reviewed May 18, 2024

View reviewed changes

lib/internal/modules/esm/get_format.js Outdated Show resolved Hide resolved

aduh95 reviewed May 18, 2024

View reviewed changes

test/parallel/test-worker-track-unmanaged-fds.js Outdated Show resolved Hide resolved

aduh95 reviewed May 18, 2024

View reviewed changes

lib/internal/modules/esm/get_format.js Outdated Show resolved Hide resolved

aduh95 reviewed May 18, 2024

View reviewed changes

lib/internal/modules/esm/get_format.js Outdated Show resolved Hide resolved

RedYetiDev mentioned this pull request May 19, 2024

Default --experimental-detect-module to commonjs in require hooks #53016

Open

dygabo force-pushed the fix-for-unflagging-module-format-detection branch 2 times, most recently from 1d7b4de to b7ec307 Compare May 20, 2024 16:42

dygabo requested review from GeoffreyBooth and aduh95 May 20, 2024 16:43

aduh95 reviewed May 20, 2024

View reviewed changes

GeoffreyBooth reviewed May 20, 2024

View reviewed changes

dygabo marked this pull request as draft May 21, 2024 12:15

dygabo added 3 commits May 26, 2024 19:34

module: resolve format for all situations with module detection on

7b99eee

review findings

3b4caab

further review findings

b2f221f

dygabo force-pushed the fix-for-unflagging-module-format-detection branch from afb82c8 to b2f221f Compare May 26, 2024 17:35

read file on cpp side if code is not available. if neither cjs nor esm

37869ec

dygabo marked this pull request as ready for review May 27, 2024 12:40

dygabo requested review from GeoffreyBooth and aduh95 May 27, 2024 12:41

lint

4e96fce

This comment was marked as off-topic.

Sign in to view

cpp lint fix

f7229d7

This comment was marked as off-topic.

Sign in to view

aduh95 reviewed May 27, 2024

View reviewed changes

test/fixtures/es-modules/require-esm-throws-with-loaders.js Outdated Show resolved Hide resolved

aduh95 reviewed May 27, 2024

View reviewed changes

test/es-module/test-esm-loader-hooks.mjs Outdated Show resolved Hide resolved

aduh95 reviewed May 27, 2024

View reviewed changes

test/es-module/test-esm-loader-hooks.mjs Outdated Show resolved Hide resolved

Apply suggestions from code review

fa9e79b

Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

module: resolve format for all situations with auto module detection on #53044

module: resolve format for all situations with auto module detection on #53044

dygabo commented May 18, 2024

nodejs-github-bot commented May 18, 2024

dygabo commented May 18, 2024

RedYetiDev commented May 18, 2024 •

edited

RedYetiDev commented May 18, 2024

RedYetiDev left a comment •

edited

GeoffreyBooth left a comment

GeoffreyBooth May 18, 2024

dygabo May 20, 2024

GeoffreyBooth May 18, 2024

RedYetiDev May 18, 2024 •

edited

dygabo May 20, 2024

dygabo commented May 18, 2024

RedYetiDev commented May 18, 2024

dygabo commented May 20, 2024

aduh95 May 20, 2024

dygabo May 21, 2024

GeoffreyBooth May 20, 2024

dygabo May 21, 2024

GeoffreyBooth May 20, 2024

dygabo May 27, 2024

GeoffreyBooth May 20, 2024

dygabo May 21, 2024

GeoffreyBooth May 20, 2024

dygabo May 21, 2024

This comment was marked as off-topic.

This comment was marked as off-topic.

aduh95 commented May 27, 2024

This comment was marked as off-topic.

This comment was marked as off-topic.

		@@ -0,0 +1,13 @@
		// Flags: --experimental-detect-module --import ./test/fixtures/es-module-loaders/builtin-named-exports.mjs

	it('should hint format correctly for extensionles modules resolve hook', async () => {
	it('should hint format correctly for the resolve hook for extensionless modules', async () => {

module: resolve format for all situations with auto module detection on #53044

Are you sure you want to change the base?

module: resolve format for all situations with auto module detection on #53044

Conversation

dygabo commented May 18, 2024

nodejs-github-bot commented May 18, 2024

dygabo commented May 18, 2024

RedYetiDev commented May 18, 2024 • edited

RedYetiDev commented May 18, 2024

RedYetiDev left a comment • edited

Choose a reason for hiding this comment

GeoffreyBooth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RedYetiDev May 18, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dygabo commented May 18, 2024

RedYetiDev commented May 18, 2024

dygabo commented May 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as off-topic.

This comment was marked as off-topic.

aduh95 commented May 27, 2024

This comment was marked as off-topic.

This comment was marked as off-topic.

RedYetiDev commented May 18, 2024 •

edited

RedYetiDev left a comment •

edited

RedYetiDev May 18, 2024 •

edited