Add local registry overlays #13926

jneem · 2024-05-17T21:14:41Z

This PR adds (private to cargo internals) support for local registry overlays, in which you can locally pretend to add packages to remote registries; the local packages will have the same source ids as the remote registry that you're overlaying.

There are two ways to set up these overlays: programmatically using GlobalContext::local_overlays and through the __CARGO_TEST_PACKAGE_CONFUSION_VULNERABILITY_DO_NOT_USE_THIS environment variable. You can't set up these overlays with .cargo/config.

The motivation for this is packaging workspaces. When we're packing a workspace, we'd like to be able to pretend (for lockfile generation and verification) that some workspace packages are already published even though they aren't.

rustbot · 2024-05-17T21:14:46Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @ehuss (or someone else) some time within the next two weeks.

Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (S-waiting-on-review and S-waiting-on-author) stays updated, invoking these commands when appropriate:

@rustbot author: the review is finished, PR author should check the comments and take action accordingly
@rustbot review: the author is ready for a review, this PR will be queued again in the reviewer's queue

Eh2406 · 2024-05-20T13:57:15Z

crates/cargo-test-support/src/lib.rs

+        if let Some(ref mut p) = self.process_builder {
+            let env_value = format!("{}={}", url, path);
+            p.env(
+                "__CARGO_TEST_PACKAGE_CONFUSION_VULNERABILITY_DO_NOT_USE_THIS",


Nit: The term of art for this kind of vulnerability is "Dependency Confusion". As with so many names in computer science, it's better to have consistency then trying to find a better name. This term was coined in the blog post https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610 which might be worth linking in comments about how users could easily miss use this.

src/cargo/sources/overlay.rs

Eh2406 · 2024-05-20T14:06:41Z

Catching Eric up on some of the context, Arlo and I discussed the questions from #10948 (comment) at office hours on Thursday. Adding this API, exclusively for internal use, seemed like the least hacky solution.

Cargo.toml

src/cargo/sources/overlay.rs

src/cargo/util/context/mod.rs

src/cargo/sources/overlay.rs

jneem · 2024-05-23T16:00:14Z

Thanks for the review! I think this is ready for another look -- we managed to successfully use this for #10948 as discussed.

tests/testsuite/registry_overlay.rs

epage · 2024-05-29T18:41:09Z

src/cargo/util/context/mod.rs

+    /// affect any newly created `PackageRegistry`s.
+    ///
+    /// See [`crate::sources::overlay::OverlayConfusionAttack`] for why you shouldn't use this.
+    pub fn local_overlays(&self) -> RefMut<'_, HashMap<SourceId, PathBuf>> {


I'd hope we can avoid putting this on GlobalContext but its hard to say without reviewing the follow up at the same time. Should we defer this to the follow up PR?

My motivation is that it gets messy adding and removing from the Global Context and if we can directly instantiate things, the easier it will be to maintain.

I can give a little more context before we decide whether or not to defer. cargo package needs the overlay to take effect during a call to ops::compile. AFAICT there are three possibilities:

put the overlay configuration in GlobalContext

put the overlay configuration in the ephemeral workspace that cargo package creates

have cargo package do something lower-level than ops::compile

The workspace option has the advantage that the overlay configuration doesn't need to be removed (because the workspace is discarded). It seems like it would just require a little work in resolve_ws and resolve_ws_with_opts to get the overlay configuration out of the workspace and insert it into the PackageRegistry

The workspace seems the most beneficial. Whichever way we go, still seems like it could be split out into a follow up PR

src/cargo/sources/config.rs

tests/testsuite/registry_overlay.rs

Cargo.toml

src/cargo/sources/config.rs

epage · 2024-06-07T18:56:35Z

btw feel free to clean up your commit history for how you want to present it for review and merge.

This adds a new mechanism for overlaying sources. An overlayed source returns packages from two (or more) sources. This functionality is not intended for public use, but it will be useful for packaging a workspace that contains inter-crate dependencies. Co-authored-by: Tor Hovland <55164+torhovland@users.noreply.github.com>

src/cargo/ops/resolve.rs

crates/cargo-test-support/src/registry.rs

Adds workspace configuration options (not user-exposed) for overlaying sources.

epage · 2024-06-11T16:25:55Z

Thanks!

@bors r+

bors · 2024-06-11T16:25:58Z

📌 Commit ba9dd1e has been approved by epage

It is now in the queue for this repository.

bors · 2024-06-11T16:27:05Z

⌛ Testing commit ba9dd1e with merge 4dcbca1...

bors · 2024-06-11T16:56:16Z

☀️ Test successful - checks-actions
Approved by: epage
Pushing 4dcbca1 to master...

rustbot assigned ehuss May 17, 2024

rustbot added A-configuration Area: cargo config files and env vars A-testing-cargo-itself Area: cargo's tests S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 17, 2024

Eh2406 reviewed May 20, 2024

View reviewed changes

src/cargo/sources/overlay.rs Outdated Show resolved Hide resolved

Eh2406 reviewed May 20, 2024

View reviewed changes

src/cargo/sources/overlay.rs Outdated Show resolved Hide resolved

Eh2406 reviewed May 20, 2024

View reviewed changes

src/cargo/sources/overlay.rs Outdated Show resolved Hide resolved

torhovland reviewed May 21, 2024

View reviewed changes

jneem mentioned this pull request May 22, 2024

[WIP] Package workspaces #13947

Draft

4 tasks

jneem requested a review from Eh2406 May 23, 2024 15:54