✨ Add experimental check for published SBOM #3903

ashearin · 2024-02-27T00:11:09Z

What kind of change does this PR introduce?

Adds check for published sboms. This PR is still a draft as there is more discussion on applicability and implementation to be had. This PR is meant to spur those conversations.

PR title follows the guidelines defined in our pull request documentation

What is the current behavior?

N/A

What is the new behavior (if this is a feature change)?**

*Tests were not added as the implementation may change, tests will be added prior to merge.

Adds a multi probe check for a Software Bill of Materials for a scanned repository.

More information regarding implementation and considerations for this check can be found here

Which issue(s) this PR fixes

Fixes #3574, #1476

Special notes for your reviewer

Does this PR introduce a user-facing change?

For user-facing changes, please add a concise, human-readable release note to
the release-note

(In particular, describe what changes users might need to make in their
application as a result of this pull request.)

Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>

…into check-for-published-sbom

codecov · 2024-02-27T00:13:57Z

Codecov Report

Merging #3903 (037eb1a) into main (d58bfb0) will decrease coverage by 4.67%.
Report is 1 commits behind head on main.
The diff coverage is 67.73%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3903      +/-   ##
==========================================
- Coverage   74.94%   70.28%   -4.67%     
==========================================
  Files         223      232       +9     
  Lines       16046    16723     +677     
==========================================
- Hits        12026    11754     -272     
- Misses       3253     4230     +977     
+ Partials      767      739      -28

laurentsimon

Thanks for this PR!

A few questions:

Should the probe be aware of cases where there's no release assets? For example, a GitHub release may have no assets, if it's only source code release. Is an SBOM needed in this case? Note that the release could contain a container release as a GitHub package or something else. We could (maybe?) update the code to detect the former case, but the latter seems much harder.
How do we determine if the project should create an SBOM or not, depending on the type of release (application, library, ?) - see https://blog.deps.dev/zillions-of-sboms/. Is there a document describing when an SBOM makes sense in each ecosystem?
Remediation depends on point (2) above. We need a good story to tell folks how to remediate. Ideally SBOMs should come for free out of the package managers, which can build enough context to decide if an SBOM is needed or not (or always generate an empty SBOM?). @steiza @di @woodruffw @JoelMarcey is this something you intend to build for npm / pypi / homebrew / rust?
Since GH has an API to generate SBOMs, why should maintainers generate an SBOM themselves?

Somewhat related but broader questions: Are registries interested in accepting SBOMs for applications. @steiza @di @woodruffw @JoelMarcey how are you thinking about this from npm / pypi / homebrew / rust side?

checker/raw_result.go

clients/githubrepo/tarball.go

probes/sbomCICDArtifactExists/impl.go

clients/gitlabrepo/sbom.go

probes/sbomCICDArtifactExists/impl.go

ashearin · 2024-02-27T18:00:43Z

@laurentsimon

Should the probe be aware of cases where there's no release assets? For example, a GitHub release may have no assets, if it's only source code release. Is an SBOM needed in this case? Note that the release could contain a container release as a GitHub package or something else. We could (maybe?) update the code to detect the former case, but the latter seems much harder.

Both clients as written handle the possibility that there are no release assets, in which case it just continues on with the remainder of the check. For a source only release I'd expect an associated SBOM, as ideally each released version would have an associated SBOM for downstream users (assuming the project should be creating an SBOM).

For releases that include a container, we could potentially check for an SBOM layer in the container, but the effort involved in that may not be worth it. This would be a situation where the workflow that builds the container in preparation for release would ideally also generate an SBOM for it and save it as an artifact (at the very least) to be picked up by our checks, and to be available for downstream users.

How do we determine if the project should create an SBOM or not, depending on the type of release (application, library, ?) - see https://blog.deps.dev/zillions-of-sboms/. Is there a document describing when an SBOM makes sense in each ecosystem?

I'm not sure about this. I know applicability is one of the sticking points for this check in general. There was talk of a Maintainers Annotation feature which could help here, but I'm not sure the status of that.

I'm unaware of any such document describing when an SBOM makes sense in each ecosystem. @idunbarh do you have any insight on this?

Since GH has an API to generate SBOMs, why should maintainers generate an SBOM themselves?

This I actually forgot to mention in the notes doc I posted, I have since added it. I do hit the SBOM api endpoint for Github repos and, assuming a good response, count that generated SBOM as a release artifact and award appropriate points. As far as generating an SBOM themselves in GH repos, I would leave it up to the maintainers to decide if that is necessary for their project. i.e. if they need to generate one for a release container or in another format. I'm hesitant to award full points for the endpoint BOM, as it may not be comprehensive. I haven't seen anything from GL yet, but would expect to see a similar SBOM endpoint eventually.

idunbarh · 2024-02-27T18:19:33Z

I'm unaware of any such document describing when an SBOM makes sense in each ecosystem. @idunbarh do you have any insight on this?

The I don't believe the OpenSSF SBOM Everywhere SIG has provided any guidance on what ecosystems should or should not generate SBOMs. The SBOM Naming Convention Doc would be the ideal place for clarification.

idunbarh · 2024-02-27T18:24:14Z

ossf/sbom-everywhere#44

Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>

laurentsimon · 2024-02-28T09:44:24Z

I'm unaware of any such document describing when an SBOM makes sense in each ecosystem. @idunbarh do you have any insight on this?

The I don't believe the OpenSSF SBOM Everywhere SIG has provided any guidance on what ecosystems should or should not generate SBOMs. The SBOM Naming Convention Doc would be the ideal place for clarification.

Yeah we really need a doc on this, per ecosystem, per-use case. What we found in https://blog.deps.dev/zillions-of-sboms/, is that most open-source projects are libraries, so often times an SBOM is not needed (there are nuances, of course). Keep me in the loop if this works starts.

laurentsimon · 2024-02-28T09:53:57Z

I'm hesitant to award full points for the endpoint BOM, as it may not be comprehensive

For a first integration, one possibility could be to not include these probes in a default check. Users who want to detect the presence of SBOMs in a repo could then run the probes / checks they care about, using the context and knowledge they have about the repo. For example, organizations (@UlisesGascon @netomi) could use their own policy to decide whether to run this probe or not on their projects.

For arbitrary projects, I think we need more investigation to reduce noise (false positives, score decrease)

pnacht · 2024-02-28T20:48:42Z

Yeah we really need a doc on this, per ecosystem, per-use case. What we found in https://blog.deps.dev/zillions-of-sboms/, is that most open-source projects are libraries, so often times an SBOM is not needed (there are nuances, of course).

Yeah, I was going to bring this up. Most projects that'll ever be handled by Scorecard likely don't need an SBOM (simple libraries).

For a first integration, one possibility could be to not include these probes in a default check.

Alternatively, make this a "bonus points" check: 10/10 if the latest release has an SBOM, $INCONCLUSIVE otherwise.

idunbarh · 2024-02-28T21:02:58Z

Coming from the Security Tooling WG discussions, one of the desired outcomes is to measure the impact other parts of OpenSSF are having around SBOM adoption. In this case the Security Tooling WG's SBOM Naming Conventions and future SBOM Strike Force that will be working directly with large projects to implement SBOMs. I see Scorecard as an mechanism to help adoption and also measure adoption.

I'm all for making it a bonus points check over a check that would be disabled by default.

Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>

…into check-for-published-sbom

spencerschrock

PR is in a generally good spot. Please address the latest review comments, and then we can merge

checker/raw_result.go

checks/evaluation/sbom.go

checks/raw/sbom.go

checks/sbom_test.go

docs/checks/internal/checks.yaml

probes/hasReleaseSBOM/impl.go

spencerschrock · 2024-05-15T19:26:15Z

After trying on github.com/bomctl/bomctl, the output can be pretty verbose. Maybe we should limit release lookback to 5 most recent similar to signed releases?

Click to expand

SCORECARD_EXPERIMENTAL=1 go run main.go --repo github.com/bomctl/bomctl --format json --show-details --checks sbom 
| jq
{
  "date": "2024-05-15T12:22:25-07:00",
  "repo": {
    "name": "github.com/bomctl/bomctl",
    "commit": "4ef4b33ce10500a853403066e335a1d48e838b11"
  },
  "scorecard": {
    "version": "devel",
    "commit": "unknown"
  },
  "score": 10.0,
  "checks": [
    {
      "details": [
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175727:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175726:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175730:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175729:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175723:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175722:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175721:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175720:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175725:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175724:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175719:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175718:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162321:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162322:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162317:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162316:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162323:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162324:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162325:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162327:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162320:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162319:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162315:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162314:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865640:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865638:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865631:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865635:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865637:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865636:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865629:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865630:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865644:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865643:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865642:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865641:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457982:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457986:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457981:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457983:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457989:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457990:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457978:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457979:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457987:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457988:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457975:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457977:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454378:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454379:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454371:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454372:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454381:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454380:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454375:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454376:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454387:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454384:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454383:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454382:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163283690:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163283686:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163283691:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163283688:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163283689:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163283687:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163123197:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163123201:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163123198:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163123200:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163123196:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163123195:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163122640:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163122637:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163122638:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163122636:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163122641:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163122635:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/162076784:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/162076780:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/162076782:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/162076783:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/162076781:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/162076778:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/153684271:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/153684275:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/153684270:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/153684272:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/153684269:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/153684273:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175727:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175726:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175730:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175729:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175723:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175722:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175721:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175720:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175725:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175724:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175719:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175718:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162321:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162322:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162317:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162316:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162323:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162324:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162325:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162327:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162320:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162319:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162315:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162314:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865640:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865638:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865631:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865635:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865637:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865636:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865629:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865630:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865644:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865643:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865642:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865641:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457982:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457986:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457981:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457983:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457989:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457990:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457978:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457979:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457987:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457988:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457975:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457977:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454378:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454379:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454371:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454372:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454381:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454380:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454375:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454376:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454387:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454384:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454383:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454382:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163283690:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163283686:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163283691:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163283688:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163283689:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163283687:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163123197:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163123201:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163123198:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163123200:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163123196:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163123195:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163122640:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163122637:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163122638:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163122636:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163122641:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163122635:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/162076784:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/162076780:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/162076782:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/162076783:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/162076781:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/162076778:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/153684271:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/153684275:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/153684270:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/153684272:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/153684269:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/153684273:0"
      ],
      "score": 10,
      "reason": "SBOM file found in release artifacts",
      "name": "SBOM",
      "documentation": {
        "url": "https://github.com/ossf/scorecard/blob/main/docs/checks.md#sbom",
        "short": "Determines if the project maintains a Software Bill of Materials."
      }
    }
  ],
  "metadata": null
}

Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>

ashearin · 2024-05-16T01:01:13Z

Same cmd with latest update. I changed it to check the 5 latest releases and only iterate over release assets until a match was found. So puts a cap on release sboms returned to 5

$ SCORECARD_EXPERIMENTAL=1 go run main.go --repo github.com/bomctl/bomctl --format json --show-details --checks sbom | jq
{
  "date": "2024-05-15T18:32:00-06:00",
  "repo": {
    "name": "github.com/bomctl/bomctl",
    "commit": "4ef4b33ce10500a853403066e335a1d48e838b11"
  },
  "scorecard": {
    "version": "devel",
    "commit": "unknown"
  },
  "score": 10.0,
  "checks": [
    {
      "details": [
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175727:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162321:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865640:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457982:0",
        "Info: Project has a SBOM file: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454378:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165175727:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/165162321:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163865640:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163457982:0",
        "Info: Project publishes an SBOM file as part of a release or CICD: https://api.github.com/repos/bomctl/bomctl/releases/assets/163454378:0"
      ],
      "score": 10,
      "reason": "SBOM file found in release artifacts",
      "name": "SBOM",
      "documentation": {
        "url": "https://github.com/ossf/scorecard/blob/main/docs/checks.md#sbom",
        "short": "Determines if the project maintains a Software Bill of Materials."
      }
    }
  ],
  "metadata": null
}

evverx · 2024-05-16T06:29:54Z

gating the check behind experimental flag

Can I ask where this is going to be used? I suspect half-baked SBOMs created to make scorecard happy combined with bogus vulnerabilities like google/osv.dev#2176 are going to waste even more maintainers' time

This PR is still a draft as there is more discussion on applicability and implementation to be had

Given that it's no longer a draft I assume it was discussed and okayed somewhere. Is it possible to take a look at those discussions anywhere?

spencerschrock · 2024-05-16T17:12:53Z

Can I ask where this is going to be used? I suspect half-baked SBOMs created to make scorecard happy combined with bogus vulnerabilities like google/osv.dev#2176 are going to waste even more maintainers' time

by people who run the scorecard binary on a repo and explicitly mention they want the SBOM check/probe.

This PR is still a draft as there is more discussion on applicability and implementation to be had

Given that it's no longer a draft I assume it was discussed and okayed somewhere. Is it possible to take a look at those discussions anywhere?

This was discussed during OSS NA 2024, there was some talk about it in the OpenSSF Slack as well. I've included some of the relevant Slack discussion below since slack history will hide it soon:

slack conversation

Dana Wang:
I met with Spencer Schrock, Laurent Simon, Ian Dunbar-Hall, Allan Friedman, Ryan Ware, justaugustus today on adding a non-opinionated check to Scorecard to report whether a project produces SBOM as the first "gold star" (Ian Dunbar-Hall owns the copyright) to report the existence of SBOM, it helps to gain visibility into the SBOM state, and put zero burden on maintainers, I feel it's a good balance on pushing the boundary on SBOM which is coming with acceleration, and protecting our maintainers. Ryan Ware has a great idea on how the Scorecard score could be tiered/applied to a project to incrementally to enhance the security posture of an open source project while protecting our maintainers from being overwhelmed with security burdens.

Brandon Mitchell:
I find myself triggered by the phrase "zero burden on maintainers". I really don't see how that's possible. Is there an example of a public project where this has been done without any maintainer involvement?

Spencer Schrock
For context, we're currently envisioning this as something that lives as a probe, not in a check. So it would not be a check in the sense of a Scorecard check, but rather as a binary "is there an SBOM".
So there would be no score attached in the normal output / score, just a "is there an SBOM?" "yes/no" when using the --format probe option.

Spencer Schrock
We're hoping this allows individual consumers to find out what their dependencies are doing, and if it matches their policy. And it may not, so the end result is "company X shouldn't use this dependency.", not that project Y should start doing Z for the benefit of company X

The consensus wasn't around including it as a default check, but rather getting some code in there for the purpose of:

Individuals being able to run the analysis if they want it (either through --checks SBOM or the corresponding --probes hasSBOM,hasReleaseSBOM)
Possibly running it as part of the weekly cron for the purpose of seeing how widely adopted SBOMs are currently. Not for the purpose of assigning a score.

As you can guess, the "default" aspect of Scorecard is the controversial part. This applies to other checks/probes as well. For example some people care about permissive license (#3838), but there's nothing inherently right or wrong with permissive vs copyleft vs other.

I've written more on this topic (which I'll be sharing "soon"), but ultimately Scorecard is a linter and should support any relevant analyses and move away from all of them being in the default output, but rather used through the enabling/disabling with individual configuration. Which lets different audiences use the tool in the manner they want without anything forced upon them.

evverx · 2024-05-16T17:35:23Z

Got it. Thank you!

people who run the scorecard binary on a repo and explicitly mention they want the SBOM check/probe

I get that part but in #3574 (and in some other issues related to SBOMs) I tried to figure out what the use cases were and I haven't seen any actual use cases anywhere.

Individuals being able to run the analysis if they want it

I wonder why? I assume this data is supposed to be acted on.

Not for the purpose of assigning a score

That's reassuring but I'm not talking about scores. I'm kind of concerned that things are seemingly getting adopted with no clear purpose and combined with OSV it offloads a lot of things onto upstream projects without any scores involved.

ultimately Scorecard is a linter and should support any relevant analyses and move away from all of them being in the default output

I agree with that.

idunbarh · 2024-05-17T18:31:27Z

🥳 Thanks @spencerschrock for all of the guidance and feedback through this PR!

ashearin added 2 commits February 22, 2024 13:45

Sbom check MVP

ff83c06

Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>

Merge branch 'main' of https://github.com/ashearin/scorecard-sandbox …

e59a2a8

…into check-for-published-sbom

ashearin requested a review from a team as a code owner February 27, 2024 00:11

ashearin requested review from justaugustus and laurentsimon and removed request for a team February 27, 2024 00:11

ashearin marked this pull request as draft February 27, 2024 00:11

ashearin temporarily deployed to gitlab February 27, 2024 00:11 — with GitHub Actions Inactive

ashearin temporarily deployed to integration-test February 27, 2024 00:11 — with GitHub Actions Inactive

ashearin added the kind/enhancement New feature or request label Feb 27, 2024

ashearin changed the title ~~Draft: ⚠️ Add Check for published SBOM~~ ⚠️ Add Check for published SBOM Feb 27, 2024

ashearin changed the title ~~⚠️ Add Check for published SBOM~~ ⚠️ Add Check for published SBOM Feb 27, 2024

ashearin self-assigned this Feb 27, 2024

ashearin added the needs discussion label Feb 27, 2024

ashearin changed the title ~~⚠️ Add Check for published SBOM~~ ✨ Add Check for published SBOM Feb 27, 2024

laurentsimon reviewed Feb 27, 2024

View reviewed changes

PR suggestion fixes

d928ede

Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>

ashearin temporarily deployed to gitlab February 27, 2024 22:02 — with GitHub Actions Inactive

ashearin temporarily deployed to integration-test February 27, 2024 22:03 — with GitHub Actions Inactive

fix line length

718d798

Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>

ashearin temporarily deployed to gitlab February 27, 2024 22:12 — with GitHub Actions Inactive

ashearin temporarily deployed to integration-test February 27, 2024 22:12 — with GitHub Actions Inactive

fix linting errors

3c99ee6

Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>

ashearin temporarily deployed to gitlab May 14, 2024 21:02 — with GitHub Actions Inactive

ashearin temporarily deployed to integration-test May 14, 2024 21:02 — with GitHub Actions Inactive

revert unnecessary changes, correct check documentation

ba44f55

Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>

ashearin temporarily deployed to gitlab May 14, 2024 21:46 — with GitHub Actions Inactive

ashearin temporarily deployed to integration-test May 14, 2024 21:46 — with GitHub Actions Inactive

Merge branch 'main' of https://github.com/ashearin/scorecard-sandbox …

413735e

…into check-for-published-sbom

ashearin temporarily deployed to gitlab May 14, 2024 22:05 — with GitHub Actions Inactive

ashearin temporarily deployed to integration-test May 14, 2024 22:05 — with GitHub Actions Inactive

ashearin marked this pull request as ready for review May 14, 2024 22:05

spencerschrock approved these changes May 15, 2024

View reviewed changes

address PR comments

1dafed9

Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>

ashearin temporarily deployed to gitlab May 15, 2024 22:15 — with GitHub Actions Inactive

ashearin temporarily deployed to integration-test May 15, 2024 22:15 — with GitHub Actions Inactive

move release lookback to data collection side

4531221

Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>

ashearin temporarily deployed to gitlab May 16, 2024 00:45 — with GitHub Actions Inactive

ashearin temporarily deployed to integration-test May 16, 2024 00:45 — with GitHub Actions Inactive

spencerschrock changed the title ~~✨ Add Check for published SBOM~~ ✨ Add experimental check for published SBOM May 17, 2024

Merge branch 'main' into check-for-published-sbom

a48f099

spencerschrock temporarily deployed to gitlab May 17, 2024 18:07 — with GitHub Actions Inactive

spencerschrock enabled auto-merge (squash) May 17, 2024 18:08

spencerschrock temporarily deployed to integration-test May 17, 2024 18:08 — with GitHub Actions Inactive

spencerschrock merged commit 8de9020 into ossf:main May 17, 2024
36 checks passed

evverx mentioned this pull request May 18, 2024

Request OSS-Fuzz project owner approval before importing OSS-Fuzz reports google/osv.dev#2176

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ Add experimental check for published SBOM #3903

✨ Add experimental check for published SBOM #3903

ashearin commented Feb 27, 2024

codecov bot commented Feb 27, 2024 •

edited

laurentsimon left a comment •

edited

ashearin commented Feb 27, 2024 •

edited

idunbarh commented Feb 27, 2024

idunbarh commented Feb 27, 2024

laurentsimon commented Feb 28, 2024

laurentsimon commented Feb 28, 2024 •

edited

pnacht commented Feb 28, 2024

idunbarh commented Feb 28, 2024

spencerschrock left a comment

spencerschrock commented May 15, 2024

ashearin commented May 16, 2024 •

edited

evverx commented May 16, 2024

spencerschrock commented May 16, 2024

evverx commented May 16, 2024

idunbarh commented May 17, 2024

✨ Add experimental check for published SBOM #3903

✨ Add experimental check for published SBOM #3903

Conversation

ashearin commented Feb 27, 2024

What kind of change does this PR introduce?

What is the current behavior?

What is the new behavior (if this is a feature change)?**

Which issue(s) this PR fixes

Special notes for your reviewer

Does this PR introduce a user-facing change?

codecov bot commented Feb 27, 2024 • edited

Codecov Report

laurentsimon left a comment • edited

Choose a reason for hiding this comment

ashearin commented Feb 27, 2024 • edited

idunbarh commented Feb 27, 2024

idunbarh commented Feb 27, 2024

laurentsimon commented Feb 28, 2024

laurentsimon commented Feb 28, 2024 • edited

pnacht commented Feb 28, 2024

idunbarh commented Feb 28, 2024

spencerschrock left a comment

Choose a reason for hiding this comment

spencerschrock commented May 15, 2024

ashearin commented May 16, 2024 • edited

evverx commented May 16, 2024

spencerschrock commented May 16, 2024

evverx commented May 16, 2024

idunbarh commented May 17, 2024

codecov bot commented Feb 27, 2024 •

edited

laurentsimon left a comment •

edited

ashearin commented Feb 27, 2024 •

edited

laurentsimon commented Feb 28, 2024 •

edited

ashearin commented May 16, 2024 •

edited