Add a new Maven pom.xml extractor #982

cuixq · 2024-05-16T01:19:42Z

The new Maven lockfile extractor aims to resolve the full Maven dependency graph to provide better transitive support #35. This is an experimental feature for now.

This PR uses deps.dev util package to parse Maven pom.xml, also calls deps.dev API for available versions when resolving a range requirement.

codecov-commenter · 2024-05-16T01:23:09Z

Codecov Report

Attention: Patch coverage is 77.50000% with 18 lines in your changes are missing coverage. Please review.

Project coverage is 64.24%. Comparing base (5eed7e8) to head (609e62d).
Report is 5 commits behind head on main.

Files	Patch %	Lines
internal/manifest/maven.go	77.50%	12 Missing and 6 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #982      +/-   ##
==========================================
+ Coverage   64.05%   64.24%   +0.18%     
==========================================
  Files         146      148       +2     
  Lines       11977    12088     +111     
==========================================
+ Hits         7672     7766      +94     
- Misses       3853     3864      +11     
- Partials      452      458       +6

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

pkg/lockfile/parse-maven-lock.go

oliverchang

LGTM with a question! Also, can you please update your PR title to reflect that this is a version of an extractor that's intended to resolve the full transitive graph?

Also reference #35 in your PR description.

oliverchang · 2024-05-17T03:55:30Z

pkg/lockfile/parse-maven-lock.go

+
+	// Otherwise return the greatest version matching the constraint.
+	// TODO: invoke Maven resolver to decide the exact version.
+	resp, err := e.Client.GetPackage(ctx, &depsdevpb.GetPackageRequest{


Question: do you think we'll ever run into cases where we need to parallelise requests to minimize latency?

I think this will be likely to happen - my plan is to make pkg/depsdev handle all requests (parallelised) to deps.dev.

G-Rath · 2024-05-17T04:02:42Z

fwiw I'm not sure that it makes sense to land this in its current form given that it's not being actually being used and violates the "offline" contract, which I don't think we can necessarily address with the current interface so landing it as-is might prematurely lock us into a public API we immediately want to change.

oliverchang · 2024-05-17T04:24:22Z

fwiw I'm not sure that it makes sense to land this in its current form given that it's not being actually being used and violates the "offline" contract, which I don't think we can necessarily address with the current interface so landing it as-is might prematurely lock us into a public API we immediately want to change.

That's a good point. Is there an easy way we can keep this interface private, while being able to use it ourselves?

Alternatively, we can just add some clear documentation to the relevant functions to say: this is experimental and can break.

oliverchang · 2024-05-20T06:31:07Z

Following up on this, the right way to do this for now is probably:

Move this to an "resolverExtractor or something similar under internal.
Modify

osv-scanner/pkg/osvscanner/osvscanner.go

Line 344 in 055ef05

func scanLockfile(r reporter.Reporter, path string, parseAs string) ([]scannedPackage, error) {

to prefer this internal resolverExtractor where it exists for a given lockfile? If --offline is passed, then we ignore these.

WDYT @cuixq @G-Rath ?

cuixq · 2024-05-21T01:30:39Z

For now, I would prefer keeping the new extractor in internal folder so we can experiment it without breaking anything.

When it's ready enough, we can think about how to enable this with the offline mode, which could be something similar to the second option.

G-Rath · 2024-05-23T03:54:43Z

fwiw if you want to enable experimental support for it without a lot of work, you should be able to add it as a custom parser here

another-rex

Thanks! mostly LGTM, added some comments/questions

another-rex · 2024-05-24T00:06:46Z

internal/manifest/helpers_test.go

+	}
+}
+
+func packageToString(pkg lockfile.PackageDetails) string {


nit: Add an example in the function comment of what the output looks like.

This is a duplicate from pkg/lockfile since I reuse the tests there...

another-rex · 2024-05-24T00:15:18Z

internal/manifest/maven.go

+		if dep.Scope != "" {
+			pkgDetails.DepGroups = append(pkgDetails.DepGroups, string(dep.Scope))
+		}
+		details[name] = pkgDetails


Is the dependency name always unique? e.g. could there be the same dependency at different versions (and recorded twice in the Dependencies array)

I'm guessing it's intentional that this is overwriting so that the last version is always the one resolved, can you add a comment a comment here clarifying this.

I would say don't worry about this too much for now since the logic will be overwritten soon (including how managed dependencies would work).

I copied these code from the current Maven extractor is sort of PoC that deps.dev Maven parser should return the same results if we use the same logic.

another-rex · 2024-05-24T00:17:53Z

internal/manifest/maven.go

+		if dep.Scope != "" {
+			pkgDetails.DepGroups = append(pkgDetails.DepGroups, string(dep.Scope))
+		}
+		details[name] = pkgDetails


Comment here as well

internal/manifest/maven.go

another-rex · 2024-05-24T00:19:54Z

internal/manifest/maven.go

+	}
+
+	// Otherwise return the greatest version matching the constraint.
+	// TODO: invoke Maven resolver to decide the exact version.


Can you clarify the difference between what it's currently doing (querying deps.dev) vs invoking the Maven resolver?

For now we only return the greatest version matching the constraint based on the versions returned by deps.dev for range requirements, however, this may not be the version that is in the dependency graph.

To figure out the exact version in the dependency graph, we need help from Maven resolver.

cuixq added 2 commits May 16, 2024 11:15

Add a new Maven lockfile extractor

2b74073

typo

349b4f7

lint

f6c7c2e

cuixq requested review from oliverchang, G-Rath, another-rex and michaelkedar May 16, 2024 01:38

oliverchang reviewed May 16, 2024

View reviewed changes

pkg/lockfile/parse-maven-lock.go Outdated Show resolved Hide resolved

rename

35f0ad8

cuixq commented May 16, 2024

View reviewed changes

pkg/lockfile/parse-maven-lock.go Outdated Show resolved Hide resolved

cuixq and others added 3 commits May 17, 2024 11:25

Merge branch 'google:main' into maven

5425d47

name

2dc15bd

name in tests

caebb9a

oliverchang approved these changes May 17, 2024

View reviewed changes

cuixq and others added 2 commits May 21, 2024 16:30

move to internal

82963d0

Merge branch 'main' into maven

9223a05

G-Rath approved these changes May 23, 2024

View reviewed changes

another-rex reviewed May 24, 2024

View reviewed changes

addressing comment

609e62d

cuixq changed the title ~~Add a new Maven lockfile extractor~~ Add a new Maven pom.xml extractor May 24, 2024

another-rex approved these changes May 24, 2024

View reviewed changes

Merge branch 'main' into maven

4d21a6e

cuixq merged commit d72f3d9 into google:main May 24, 2024
12 of 13 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a new Maven pom.xml extractor #982

Add a new Maven pom.xml extractor #982

cuixq commented May 16, 2024 •

edited

codecov-commenter commented May 16, 2024 •

edited

oliverchang left a comment

oliverchang May 17, 2024

cuixq May 17, 2024

G-Rath commented May 17, 2024

oliverchang commented May 17, 2024

oliverchang commented May 20, 2024

cuixq commented May 21, 2024

G-Rath commented May 23, 2024

another-rex left a comment •

edited

another-rex May 24, 2024

cuixq May 24, 2024

another-rex May 24, 2024

cuixq May 24, 2024

another-rex May 24, 2024

another-rex May 24, 2024

cuixq May 24, 2024

Add a new Maven pom.xml extractor #982

Add a new Maven pom.xml extractor #982

Conversation

cuixq commented May 16, 2024 • edited

codecov-commenter commented May 16, 2024 • edited

Codecov Report

oliverchang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

G-Rath commented May 17, 2024

oliverchang commented May 17, 2024

oliverchang commented May 20, 2024

cuixq commented May 21, 2024

G-Rath commented May 23, 2024

another-rex left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cuixq commented May 16, 2024 •

edited

codecov-commenter commented May 16, 2024 •

edited

another-rex left a comment •

edited