Mageia vulnerabilities available in OSV

[dfandrich]

The Mageia distribution now exports its vulnerability reports in OSV format. Here are some key URLs:

Source URL: https://advisories.mageia.org/<ID>.html
OSV Formatted URL: https://advisories.mageia.org/<ID>.json
Index of vulnerabilities: https://advisories.mageia.org/vulns.json
Mageia security advisories home: https://advisories.mageia.org/

We're using a PURL format analogous to the Fedora one. The index format is compatible with the one used by Go, except that the advisory modification time is not currently easily accessible in our infrastructure so it's left off.

I'm not sure what the most relevant link is for "How to contribute", but our Bugzilla instance is at https://bugs.mageia.org/ and we have a wiki page on ways to contribute to the distribution at https://wiki.mageia.org/en/Contributing.

[oliverchang]

Thanks @dfandrich ! https://advisories.mageia.org/vulns.json doesn't seem to work at the moment. Is this expected to be live soon?

Also, would you be able to contribute an OSV schema definition here: https://ossf.github.io/osv-schema/#affectedpackage-field to define the ecosystem/package naming rules?

[dfandrich]

I think you checked that URL during the time our servers were down due to a cooling issue in the datacentre. I can create a PR on the schema definition.

[dfandrich]

I've created ossf/osv-schema#235

[dfandrich]

I've also created #2107 on source_test.yaml. That one probably isn't useful right now without changes to the code, but it's a starting point for discussion. All but two sources (that return all advisories in a single giant file) seem to use a cloud storage directory API to retrieve a list. Mageia currently has a REST endpoint to get a list of IDs, then each desired one must be retrieved in turn.

[andrewpollock]

Leaving some notes here for future reference:

• https://advisories.mageia.org/vulns.json emits vulnerability IDs
• https://advisories.mageia.org/MGASA-2024-0129.json emits a single vulnerability in OSV format

Comparing with the two existing REST sources:

• https://curl.se/docs/vuln.json
• https://packages.wolfi.dev/os/osv/all.json

which emit an array of all the vulnerabilities.

@dfandrich how difficult would it be to stand up another endpoint that in essence

• fetches all the IDs from https://advisories.mageia.org/vulns.json
• iterates over them building up an array
• emits the array
  ?

[dfandrich]

It wouldn't be difficult to set up an endpoint for all advisories, but it's not something I would want to do—it's just not scalable. We support about 10,000 packages, which means that we publish up to 10,000 times more advisories than any average single project would. All our advisories (including bug fix advisories) would take around 27 MiB right now if they were in a single file. We don't really want dozens (or hundreds or tens of thousands) of OSV users hammering our lone server to download the full bundle several times a day just to see if there's anything new, especially since there usually IS something new every day. All the other OSV sources offer an index, even if that index is provided by Google Cloud Storage, despite most also being front-ended by a CDN so they don't even need to care about bandwidth. I think offering an index to the advisories, like almost all other OSV sources, is a reasonable way to provide access to all of them. If there were a OSV-defined standard for such an index, I'd even be happy to switch to that.

[oliverchang]

An index is totally fine for the main vulnerability JSON, and is supported per https://google.github.io/osv.dev/rest-api-contribution/#1-a-url-pointing-to-a-rest-endpoint-containing-at-least-all-of-the-vulnerabilities-ids-and-date-modified.

The only change we'd like to see is the addition of modified in https://advisories.mageia.org/vulns.json -- would this be feasible?

[dfandrich]

The modified timestamp is not currently available in our advisory publishing workflow, but it is something that we could add without huge difficulty, if necessary. I didn't spot that page documenting the index file before.

[oliverchang]

It is indeed necessary for our import process to work. Would you be able to add it?

[dfandrich]

I went ahead and added it this morning. It's live now: https://advisories.mageia.org/vulns.json

[andrewpollock]

I didn't spot that page documenting the index file before.

Hi @dfandrich if you have any feedback on our documentation or on your user journey navigating it, I'm all ears. Our new data source onboarding process is very bumpy, manual and bespoke right now, and while I don't foresee OSV.dev's data sources growing at the same rate or to the same scale as the CVE Program's CNA's, that could also be famous last words...

So, good quality, easily navigable documentation (and a soon to be created checklist with concrete examples) are the only way to smoothly scale here :-)

[dfandrich]

My main source of confusion about the process is that the information I needed was spread out about several web sites & repositories and it was hard to find all the information I needed. I couldn't find the specification on the JSON index format until it was pointed out to me, and the same with the source.yaml file (and I still haven't found documentation on that one). It also seemed a bit odd to me that the OSV schema specification includes information about the data sources themselves, although I suppose the prefixes do fit. Even now, it's not completely clear to me the scope of https://osv.dev/ and how that web site and API fits in to the whole OSV "ecosystem" if you want to use that term.

[andrewpollock]

Hi @dfandrich the new home database onboarding process is far from streamlined (for the home database or for us). If you're up for giving me a bit of a brain dump while things are still fresh in your mind, I'm all ears. My goal is to produce a checklist with real world example PRs to crib from, at a minimum.

[dfandrich]

Sure, I'm happy to help. Let me know what I can do.

[andrewpollock]

What time zone are you in? It's probably going to be best to talk through your experiences interactively.

[dfandrich]

I'm in UTC+7. Feel free to send me an e-mail to set something up.