-
Notifications
You must be signed in to change notification settings - Fork 450
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Scorecard should provide details on fuzzing coverage in fuzzing check #78
Comments
I think it would be good to understand fuzzer health overall. So, coverage is a big part, but so is performance, whether issues get fixed, other efficacy metrics that OSS-fuzz emits. If there was a way to capture this in simple A/B/C grades or the like it would be very useful. CC @asraa |
@laurentsimon also mentioned this idea, adding him as fyi |
@inferno-chromium does oss-fuzz provide those details? |
They're up on GCS for each project's latest coverage report e.g. I'm not sure that's the easiest way to get them |
This a good start. Thanks. I will work on updating the fuzzing with that information. |
Is there a json result? I don't have permission to access the dir. We can use the HTTP endpoint to query the results and update the fuzzing results. |
@inferno-chromium / @asraa Ping. |
@oliverchang can help on providing the json endpoint/gcs info here. |
Sure! So to get the coverage for a OSS-Fuzz project, you need to read 2 JSON files. First from e.g. https://oss-fuzz-coverage.storage.googleapis.com/latest_report_info/woff2.json Then get the That JSON will have a |
The fuzzing results provide all these details. type ossfuzz struct {
Data []struct {
Files []struct {
Filename string `json:"filename"`
Summary struct {
Branches struct {
Count int `json:"count"`
Covered int `json:"covered"`
Notcovered int `json:"notcovered"`
Percent int `json:"percent"`
} `json:"branches"`
Functions struct {
Count int `json:"count"`
Covered int `json:"covered"`
Percent int `json:"percent"`
} `json:"functions"`
Instantiations struct {
Count int `json:"count"`
Covered int `json:"covered"`
Percent int `json:"percent"`
} `json:"instantiations"`
Lines struct {
Count int `json:"count"`
Covered int `json:"covered"`
Percent int `json:"percent"`
} `json:"lines"`
Regions struct {
Count int `json:"count"`
Covered int `json:"covered"`
Notcovered int `json:"notcovered"`
Percent int `json:"percent"`
} `json:"regions"`
} `json:"summary"`
} `json:"files"`
Totals struct {
Branches struct {
Count int `json:"count"`
Covered int `json:"covered"`
Notcovered int `json:"notcovered"`
Percent float64 `json:"percent"`
} `json:"branches"`
Functions struct {
Count int `json:"count"`
Covered int `json:"covered"`
Percent int `json:"percent"`
} `json:"functions"`
Instantiations struct {
Count int `json:"count"`
Covered int `json:"covered"`
Percent float64 `json:"percent"`
} `json:"instantiations"`
Lines struct {
Count int `json:"count"`
Covered int `json:"covered"`
Percent float64 `json:"percent"`
} `json:"lines"`
Regions struct {
Count int `json:"count"`
Covered int `json:"covered"`
Notcovered int `json:"notcovered"`
Percent float64 `json:"percent"`
} `json:"regions"`
} `json:"totals"`
} `json:"data"`
Type string `json:"type"`
Version string `json:"version"`
} Questions
type CheckResult struct {
Error error `json:"-"`
Name string
Details []string
Confidence int
Pass bool
ShouldRetry bool `json:"-"`
} We could extend the results by changing |
Totals -> Regions -> Percent should be good enough for now (high-level % of code covered). Additional details we can later add once we have some design on the Details string as you said. Right now, too many other things to worry about, but can come back in a month to create maybe some structure on Details field ? @azeemshaikh38 thoughts? |
+1 to adding this to Details []string for now. I think this is another of those things which will benefit from implementing #347 (comment). We should probably consider updating how Details []string looks like after few weeks. But for now, there are too many moving parts and I don't want to add one more. |
We could wait for the If we implement with @inferno-chromium Is this an issue that can wait ? are there teams waiting for this to be addressed? |
+1 to waiting if this is not a blocker. |
@azeemshaikh38 this would be a good feature to provide more details for package consumers. Can the result interface accommodate some of these changes? |
Possible ideas:
(1) is the simplest to start with, and can be improved later to support a better solution. |
Discussed in 5/16 meeting: nice to have, but unsure if this functionality could be extended to non-ossfuzz fuzzers |
@htuch's suggestion.
The text was updated successfully, but these errors were encountered: