Fuzzing Coverage Expansion #866

viktoriia-lsg · 2023-09-25T16:57:21Z

This pull requests expands fuzzing coverage for OSS-fuzz by adding 4 more fuzz test files and 5 corpora files. All fuzz tests are written in native go fuzzing style. Corpora files are compatible with OSS-fuzz format.

New fuzzers and corpora were added for OSS-Fuzz.

fse/fse_fuzz_test.go

huff0/huff0_fuzz_test.go

s2/s2_fuzz_test.go

snappy/xerial/xerial_encode_fuzz_test.go

huff0/huff0_fuzz_test.go

klauspost · 2023-09-26T08:36:43Z

Please see #289 for reference.

A) I would like to move seed files off-repo, since it is making the repo too large. I have an old repo that was used for the old fuzz tests, but can be repurposed for this.

B) I do not want to end up like with #289 - where a third party controls scripts that I cannot modify freely. This means I have been unable to modify/add my fuzz tests for half a year now. I have asked @AdamKorcz for assistance in this, but no response. If I am not in control of the setup, I will not accept this.

Needless to say I am not very happy about the current setup and would like that fixed as a primary thing, which TBH is more important than adding coverage.

If you need assistance in making CI pass, let me know.

Renaming from x_fuzz_test.go to fuzz_test.go

Fixes `panic: runtime error: index out of range [7] with length 5` when providing short input to EstimateBlockSize. Found via #866

viktoriia-lsg · 2023-09-26T12:53:10Z

Regarding fuzzing setup:
A) I think it can be done easily. I just need to rewrite configuration file for it to take files from the github repository. So it's not a problem.
B) As the configuration file (build.sh) is located on oss-fuzz repo, I still need to make PR to update it. To satisfy your needs, we can change build.sh to the following format, so the oss-fuzz will take build script from your repo:

$SRC/compress/ossfuzz.sh

Does it make sense?

klauspost · 2023-09-26T13:28:26Z

Seems a file is missing:

--- FAIL: FuzzEncoder (0.00s)
    helpers.go:36: open testdata/fuzz/block-corpus-raw.zip: no such file or directory
FAIL

workaround in s2 is removed; snappy fuzzer is moved to fuzz_test; missing file is added.

viktoriia-lsg · 2023-09-26T13:45:44Z

Thanks! I've added a missing file and slightly changed a path to it.

klauspost · 2023-09-26T14:22:48Z

Run diff <(gofmt -d .) <(printf "")
1,13d0
< diff fse/fuzz_test.go.orig fse/fuzz_test.go
< --- fse/fuzz_test.go.orig
< +++ fse/fuzz_test.go
< @@ -3,[8](https://github.com/klauspost/compress/actions/runs/6313933219/job/17143098604?pr=866#step:4:9) +3,8 @@
<  import (
<  	"bytes"
<  	"fmt"
< -	"testing"
<  	"github.com/klauspost/compress/internal/fuzz"
< +	"testing"
<  )
<

viktoriia-lsg · 2023-09-26T14:38:36Z

Does it mean that fse/fuzz_test.go needs a newline after "testing" package?
Regarding s2_fuzz_test.go file, is everything fine, so I can merge it with s2/fuzz_test.go? Fuzzing Coverage Expansion #866 (comment)
What do you think about this setup? Fuzzing Coverage Expansion #866 (comment)

klauspost · 2023-09-29T07:52:27Z

Does it mean that fse/fuzz_test.go needs a newline after "testing" package?

Imports are always sorted.

Regarding s2_fuzz_test.go file, is everything fine, so I can merge it with s2/fuzz_test.go? Fuzzing Coverage Expansion #866 (comment)

Yes, please do that.

What do you think about this setup? Fuzzing Coverage Expansion #866 (comment)

Sounds like a reasonable approach.

Fix import issue in fse; Merge s2_fuzz_test with s2/fuzz_test.

Moving setup_dicts.go from oss-fuzz

viktoriia-lsg · 2023-09-29T17:25:19Z

Alright, we started working on it.

Setup_dicts is moved to ossfuzz dir; Configuration file ossfuzz.sh is moved to repo.

Adding a comment

viktoriia-lsg · 2023-10-03T12:57:51Z

Hi @klauspost! We've made changes as discussed earlier. We've also opened a pull request to compress-fuzz with seed corpuses klauspost/compress-fuzz#2.

Please let us know, if we need to change anything.

klauspost · 2023-10-03T13:01:23Z

Thanks! I will run the tests and give it a final review.

klauspost

Just one change and a question.

snappy/xerial/testdata/block-corpus-raw.zip

ossfuzz/ossfuzz.sh

viktoriia-lsg · 2023-10-03T15:44:20Z

I have edited ossfuzz.sh file and decreased the seed corpus in size.

klauspost · 2023-10-03T16:04:53Z

Looks good!

viktoriia-lsg · 2023-10-03T16:07:57Z

Thanks! Please also review klauspost/compress-fuzz#2 , so I can make a PR to oss-fuzz in order to start fuzzing.

klauspost · 2023-10-03T16:11:14Z

Thanks! Please also review klauspost/compress-fuzz#2 , so I can make a PR to oss-fuzz in order to start fuzzing.

Ah. Somehow I missed that. Will take a look - probably tomorrow.

This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [github.com/klauspost/compress](https://togithub.com/klauspost/compress) | indirect | patch | `v1.17.0` -> `v1.17.2` | --- ### Release Notes <details> <summary>klauspost/compress (github.com/klauspost/compress)</summary> ### [`v1.17.2`](https://togithub.com/klauspost/compress/releases/tag/v1.17.2) [Compare Source](https://togithub.com/klauspost/compress/compare/v1.17.1...v1.17.2) #### What's Changed - zstd: Fix corrupted output in "best" by [@klauspost](https://togithub.com/klauspost) in [klauspost/compress#876 **Full Changelog**: klauspost/compress@v1.17.1...v1.17.2 ### [`v1.17.1`](https://togithub.com/klauspost/compress/releases/tag/v1.17.1) [Compare Source](https://togithub.com/klauspost/compress/compare/v1.17.0...v1.17.1) #### What's Changed - s2: Fix S2 "best" dictionary wrong encoding by [@klauspost](https://togithub.com/klauspost) in [klauspost/compress#871 - flate: Reduce allocations in decompressor and minor code improvements by [@fakefloordiv](https://togithub.com/fakefloordiv) in [klauspost/compress#869 - s2: Fix EstimateBlockSize on 6&7 length input by [@klauspost](https://togithub.com/klauspost) in [klauspost/compress#867 - tests: Fuzzing Coverage Expansion by [@viktoriia-lsg](https://togithub.com/viktoriia-lsg) in [klauspost/compress#866 - tests: Set FSE decompress fuzzer max limit by [@klauspost](https://togithub.com/klauspost) in [klauspost/compress#868 - tests: Fuzzing Coverage Expansion ([#2](https://togithub.com/klauspost/compress/issues/2)) by [@viktoriia-lsg](https://togithub.com/viktoriia-lsg) in [klauspost/compress#870 #### New Contributors - [@viktoriia-lsg](https://togithub.com/viktoriia-lsg) made their first contribution in [klauspost/compress#866 - [@fakefloordiv](https://togithub.com/fakefloordiv) made their first contribution in [klauspost/compress#869 **Full Changelog**: klauspost/compress@v1.17.0...v1.17.1 </details> --- ### Configuration 📅 **Schedule**: Branch creation - "before 4am on the first day of the month" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).

New fuzzers and corpora were added

b4a97aa

New fuzzers and corpora were added for OSS-Fuzz.

klauspost requested changes Sep 26, 2023

View reviewed changes

Fuzz tests files were renamed

fd889b2

Renaming from x_fuzz_test.go to fuzz_test.go

klauspost added a commit that referenced this pull request Sep 26, 2023

s2: Fix EstimateBlockSize on 6&7 length input

15010fc

Fixes `panic: runtime error: index out of range [7] with length 5` when providing short input to EstimateBlockSize. Found via #866

klauspost mentioned this pull request Sep 26, 2023

s2: Fix EstimateBlockSize on 6&7 length input #867

Merged

klauspost added a commit that referenced this pull request Sep 26, 2023

s2: Fix EstimateBlockSize on 6&7 length input (#867)

e670df9

Fixes `panic: runtime error: index out of range [7] with length 5` when providing short input to EstimateBlockSize. Found via #866

Minor changes in fuzzers

a3537ac

workaround in s2 is removed; snappy fuzzer is moved to fuzz_test; missing file is added.

Merge branch 'master' into master

5e5fcbb

viktoriia-lsg and others added 4 commits September 29, 2023 14:24

Minor changes in fuzzers

6674dce

Fix import issue in fse; Merge s2_fuzz_test with s2/fuzz_test.

Merge branch 'klauspost:master' into test

d0b7c34

Create setup_dicts.go

503203c

Moving setup_dicts.go from oss-fuzz

Moving setup_dicts.go

323658a

viktoriia-lsg added 3 commits October 3, 2023 14:14

Creating ossfuzz dir with configs

cf75ee2

Setup_dicts is moved to ossfuzz dir; Configuration file ossfuzz.sh is moved to repo.

Update ossfuzz.sh

f580642

Adding a comment

Update ossfuzz.sh permissions

b98a1a2

viktoriia-lsg mentioned this pull request Oct 3, 2023

Fuzzing Coverage Expansion (moving seed corpuses) klauspost/compress-fuzz#2

Merged

Merge branch 'test'

4dcb711

klauspost requested changes Oct 3, 2023

View reviewed changes

snappy/xerial/testdata/block-corpus-raw.zip Outdated Show resolved Hide resolved

ossfuzz/ossfuzz.sh Outdated Show resolved Hide resolved

viktoriia-lsg added 2 commits October 3, 2023 17:40

Comment out fork of go-fuzz-headers

81eaec6

Minor changes in ossfuzz.sh and seed corpus size

ba0e5f1

Merge branch 'test'

6548538

klauspost approved these changes Oct 3, 2023

View reviewed changes

klauspost merged commit 0e1030c into klauspost:master Oct 4, 2023
18 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fuzzing Coverage Expansion #866

Fuzzing Coverage Expansion #866

viktoriia-lsg commented Sep 25, 2023

klauspost commented Sep 26, 2023 •

edited

viktoriia-lsg commented Sep 26, 2023

klauspost commented Sep 26, 2023

viktoriia-lsg commented Sep 26, 2023

klauspost commented Sep 26, 2023

viktoriia-lsg commented Sep 26, 2023

klauspost commented Sep 29, 2023

viktoriia-lsg commented Sep 29, 2023

viktoriia-lsg commented Oct 3, 2023

klauspost commented Oct 3, 2023

klauspost left a comment

viktoriia-lsg commented Oct 3, 2023

klauspost commented Oct 3, 2023

viktoriia-lsg commented Oct 3, 2023

klauspost commented Oct 3, 2023

Fuzzing Coverage Expansion #866

Fuzzing Coverage Expansion #866

Conversation

viktoriia-lsg commented Sep 25, 2023

klauspost commented Sep 26, 2023 • edited

viktoriia-lsg commented Sep 26, 2023

klauspost commented Sep 26, 2023

viktoriia-lsg commented Sep 26, 2023

klauspost commented Sep 26, 2023

viktoriia-lsg commented Sep 26, 2023

klauspost commented Sep 29, 2023

viktoriia-lsg commented Sep 29, 2023

viktoriia-lsg commented Oct 3, 2023

klauspost commented Oct 3, 2023

klauspost left a comment

Choose a reason for hiding this comment

viktoriia-lsg commented Oct 3, 2023

klauspost commented Oct 3, 2023

viktoriia-lsg commented Oct 3, 2023

klauspost commented Oct 3, 2023

klauspost commented Sep 26, 2023 •

edited