Chokidar does not handle kFSEventStreamEventFlagMustScanSubDirs properly, calls unlink callback

[MarcCelani-at]

Describe the bug

If the system is flooded with too many fsevents (for example, you are watching a directory that undergoes a huge number of changes all at once, and the user code to process these events is slow), a buffer of events can fill up and trigger a kFSEventStreamEventFlagUserDropped event:

The kFSEventStreamEventFlagUserDropped or kFSEventStreamEventFlagKernelDropped flags may be set in addition to the kFSEventStreamEventFlagMustScanSubDirs flag to indicate that a problem occurred in buffering the events (the particular flag set indicates where the problem occurred) and that the client must do a full scan of any directories (and their subdirectories, recursively) being monitored by this stream. If you asked to monitor multiple paths with this stream then you will be notified about all of them. Your code need only check for the kFSEventStreamEventFlagMustScanSubDirs flag; these flags (if present) only provide information to help you diagnose the problem.

When this happens...

fsevents calls the watch callback on the root directory with flags=3 (kFSEventStreamEventFlagMustScanSubDirs & kFSEventStreamEventFlagUserDropped), and then calls the callback for every file and subdirectory.

When chokidar processes the event on the root directory, it calls fsevents.getInfo and compares the type to the previous type. If the new type does not match the old type, it calls the callback with the unlink event.

Chokidar code:
https://github.com/paulmillr/chokidar/blob/master/lib/fsevents-handler.js#L151
https://github.com/paulmillr/chokidar/blob/master/lib/fsevents-handler.js#L331-L335

The problem is that fsevents.getInfo().type will be undefined if the flags=3. (Relevant: fsevents/fsevents#372) See the implementation here: https://github.com/fsevents/fsevents/blob/master/fsevents.js#L40-L54

Because "directory" !== undefined, chokidar incorrectly calls the unlink event on the root directory, and user applications cannot distinguish between a true unlink and the kFSEventStreamEventFlagMustScanSubDirs event.

Versions (please complete the following information):

• Chokidar version: 3.5.2
• Node version: 12.20.1
• OS version: MacOS 12.1

To Reproduce:

Create a chokidar watch on a directory. The callback should do something expensive like busy-loop. Flood the directory with frequent updates to a large number of files.

Expected behavior
Chokidar sends a special event signaling that this behavior has happened, or at very least does not call unlink on the root directory.

Additional context
We are working on a large mono repo and watch the root directory to transpile our code. We have a separate tool that typechecks the entire repo and outputs .d.ts files to a .gitignored directory within the repo. When running these tools at the same time, typescript generates a large number of watches that are somewhat expensive to process (need to re-transpile the code) which repros this issue.

-Submitted by Airtable

[MarcCelani-at]

@paulmillr the way we are handling this is to just drop events with flag kFSEventStreamEventFlagMustScanSubDirs. fsevents is calling the callback for all of the subdirs and files within the root directory after this, so that seems relatively safe. If that approach makes sense to you, I can send the patch we applied as a PR. I probably won't have time to figure out how to write a unit test for this, though.

[paulmillr]

Sure

[davidfirst]

I was debugging this for a while and got to the same conclusions.
In my case, I got this flags 3 when deleting a cache directory with lots of files and directories. I set this dir in the "ignore" options, but its rootDir was watched.
For some reason, this happens only when deleting the directory async (with fs.remove, not fs.removeSync) from node. Deleting from the cli doesn't trigger this flag.
It starts by firing a few flags=131584 (deleted directory) and then flags=3 with the root-dir.

As a result, as mentioned above, the root-dir got the "unlink" event by Chokidar, and then Chokidar fired unlink events for all its files and sub directories.

Mac OS 12.6
Node v18.16.1

@paulmillr , any chance to merge the PR above that fixes it? it'd be a great help!
Thanks.