Skip to content

Commit

Permalink
docs: Refresh daemon documentation (#7386)
Browse files Browse the repository at this point in the history
### Description

The daemon documentation was a little out of date. I added some docs
about the architecture. Also added some docs to cookies since they're
not a commonly known concept.

### Testing Instructions

<!--
  Give a quick description of steps to test your changes.
-->


Closes TURBO-2364
  • Loading branch information
NicholasLYang committed Feb 15, 2024
1 parent 509a4cb commit c77bfc6
Show file tree
Hide file tree
Showing 3 changed files with 65 additions and 11 deletions.
41 changes: 41 additions & 0 deletions crates/turborepo-filewatch/src/cookies.rs
@@ -1,3 +1,38 @@
//! Cookies are the file watcher's way of synchronizing file system events. They
//! are files that are added to the file system that are named with the format
//! `[id].cookie`, where `[id]` is an increasing serial number, e.g.
//! `1.cookie`, `2.cookie`, and so on. The daemon can then watch for the
//! file creation event for this cookie file. Once it sees this event,
//! the daemon knows that the file system events are up to date and we
//! won't get any stale events.
//!
//! Here's the `CookieWriter` flow:
//! - `CookieWriter` spins up a `watch_cookies` task and creates a
//! `cookie_requests` mpsc channel to send a cookie request to that task. The
//! cookie request consists of a oneshot `Sender` that the task can use to
//! send back the serial number.
//! - The `watch_cookies` task watches for cookie requests on
//! `cookie_requests_rx`. When one occurs, it creates the cookie file and
//! bumps the serial. It then sends the serial back using the `Sender`
//! - When `CookieWriter::cookie_request` is called, it sends the cookie request
//! to the `watch_cookies` channel and then waits for the serial as a response
//! (with a timeout). Upon getting the serial, a `CookiedRequest` gets
//! returned with the serial number attached.
//!
//! And here's the `CookieWatcher` flow:
//! - `GlobWatcher` creates a `CookieWatcher`.
//! - `GlobWatcher` gets queries about globs that are wrapped in
//! `CookiedRequest`. It passes these requests to
//! `CookieWatcher::check_request`
//! - If the serial number attached to `CookiedRequest` has already been seen,
//! `CookieWatcher::check_request` returns the inner query immediately.
//! Otherwise, it gets stored in `CookieWatcher`.
//! - `GlobWatcher` waits for file system events on `recv`. When it gets an
//! event, it passes the event to `CookieWatcher::pop_ready_requests`. If this
//! event is indeed a cookie event, we return all of the requests that are now
//! allowed to be processed (i.e. their serial number is now less than or
//! equal to the latest seen serial).

use std::{collections::BinaryHeap, fs::OpenOptions, time::Duration};

use notify::EventKind;
Expand Down Expand Up @@ -38,6 +73,8 @@ pub struct CookieWriter {
_exit_ch: mpsc::Sender<()>,
}

/// A request that can only be processed after the `serial` has been seen by the
/// `CookieWatcher`.
#[derive(Debug)]
pub struct CookiedRequest<T> {
request: T,
Expand Down Expand Up @@ -158,6 +195,10 @@ impl CookieWriter {
&self.root
}

/// Sends a request to make a cookie file to the
/// `watch_for_cookie_file_requests` task. Waits on a response from the
/// task, and returns a `CookiedRequest` with the expected serial
/// number.
pub(crate) async fn cookie_request<T>(
&self,
request: T,
Expand Down
22 changes: 22 additions & 0 deletions crates/turborepo-lib/src/daemon/mod.rs
@@ -1,3 +1,25 @@
//! The Turborepo daemon watches files and pre-computes data to speed up turbo's
//! execution. Each repository has a separate daemon instance.
//!
//! # Architecture
//! The daemon consists of a gRPC server that can be queried by a client.

//! The server spins up a `FileWatching` struct, which contains a struct
//! responsible for watching the repository (`FileSystemWatcher`), and the
//! various consumers of that file change data such as `GlobWatcher` and
//! `PackageWatcher`.
//!
//! We use cookie files to ensure proper event synchronization, i.e.
//! that we don't get stale file system events while handling queries.
//!
//! # Naming Conventions
//! `recv` is a receiver of file system events. Structs such as `GlobWatcher`
//! or `PackageWatcher` consume these file system events and either derive state
//! or produce new events.
//!
//! `_tx`/`_rx` suffixes indicate that this variable is respectively a `Sender`
//! or `Receiver`.

mod bump_timeout;
mod bump_timeout_layer;
mod client;
Expand Down
13 changes: 2 additions & 11 deletions crates/turborepo-lib/src/daemon/server.rs
@@ -1,16 +1,7 @@
//! Daemon Server
//!
//! This module houses the daemon server, some implementation notes for which
//! are below.
//!
//! ## Implementation Notes
//!
//! The basic goals of the daemon are to watch for, and be able to provide
//! details about, filesystem changes. It is organised as an async server, which
//! holds a `HashGlobWatcher` which holds data about hashes, globs to watch for
//! that hash, and files that have been updated for that hash. In addition, this
//! server can be interrogated over grpc to register interest in particular
//! globs, and to query for changes for those globs.
//! This module houses the daemon server. For more information, go to the
//! [daemon module](std::daemon).

use std::{
collections::{HashMap, HashSet},
Expand Down

0 comments on commit c77bfc6

Please sign in to comment.