Skip to content

Commit

Permalink
fuzz: initial integration
Browse files Browse the repository at this point in the history
 - Adds initial fuzz_target for Handlebars.compile()
 - Adds Jazzer.js as dev-dependency
 - Adds fuzzing on Github CI
  • Loading branch information
manunio committed Dec 6, 2023
1 parent 1fc4ef0 commit f1fffb4
Show file tree
Hide file tree
Showing 62 changed files with 2,878 additions and 414 deletions.
4 changes: 4 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,10 @@ jobs:
cd ./tests/integration/webpack-babel-test && ./test.sh && cd -
cd ./tests/integration/webpack-test && ./test.sh && cd -
- name: Fuzz Test
# Runs for 2 minutes
run: cd ./fuzz && ./test.sh 120 && cd -

browser:
name: Test (Browser)
runs-on: 'ubuntu-22.04'
Expand Down
45 changes: 45 additions & 0 deletions fuzz/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# Fuzz Testing

Fuzz testing is:

> An automated software testing technique that involves providing invalid, unexpected, or random data as inputs to a program.
We use coverage guided fuzz testing to automatically discover bugs in Handlebars.js.

This `fuzz/` directory contains the configuration and the fuzz tests for Handlebars.js.
To generate and run fuzz tests, we use the [Jazzer.js](https://github.com/CodeIntelligenceTesting/jazzer.js/) library.

## Running a fuzzer

This directory contains fuzzers like for example `compiler.fuzz`. You can run it with:

```sh
$ npx jazzer fuzz/compiler.fuzz fuzz/corpus/compiler.fuzz/ --sync
```

You should see output that looks something like this:

```
#7 INITED cov: 20 ft: 20 corp: 1/4b exec/s: 0 rss: 162Mb
#20 REDUCE cov: 20 ft: 20 corp: 1/3b lim: 4 exec/s: 0 rss: 162Mb L: 3/3 MS: 3 CopyPart-ChangeBit-EraseBytes-
#45 REDUCE cov: 20 ft: 20 corp: 1/2b lim: 4 exec/s: 0 rss: 162Mb L: 2/2 MS: 5 CrossOver-ChangeByte-CopyPart-CopyPart-EraseBytes-
#46 REDUCE cov: 20 ft: 20 corp: 1/1b lim: 4 exec/s: 0 rss: 162Mb L: 1/1 MS: 1 EraseBytes-
#219 REDUCE cov: 25 ft: 25 corp: 2/4b lim: 4 exec/s: 0 rss: 162Mb L: 3/3 MS: 3 ChangeBit-ChangeBit-CMP- DE: "\001\000"-
#220 REDUCE cov: 25 ft: 25 corp: 2/3b lim: 4 exec/s: 0 rss: 162Mb L: 2/2 MS: 1 EraseBytes-
#293 REDUCE cov: 25 ft: 25 corp: 2/2b lim: 4 exec/s: 0 rss: 162Mb L: 1/1 MS: 3 ChangeByte-ShuffleBytes-EraseBytes-
```

It will continue to generate random inputs forever, until it finds a
bug or is terminated. The testcases for bugs it finds can be seen in
the form of `crash-*` or `timeout-*` at the place from where command is run.
You can rerun the fuzzer on a single input by passing it on the
command line `npx jazzer fuzz/compiler.fuzz /path/to/testcase`.

## The corpus

The corpus is a set of test inputs, stored as individual files,
provided to the fuzz target as a starting point (to “seed” the mutations).
The quality of the seed corpus has a huge impact on fuzzing efficiency;
the higher the quality, the easier it is for the fuzzer to discover new code paths.
The ideal corpus is a minimal set of inputs that provides maximal code coverage
Each fuzzer has an individual corpus under `fuzz/corpus/test_name`.
27 changes: 27 additions & 0 deletions fuzz/compiler.fuzz.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
const Handlebars = require('../dist/cjs/handlebars')['default'];

const ignored = [
'Parse error',
'Lexical error',
`doesn't match`,
'Invalid path',
'Unsupported number of partial arguments',
];

function ignoredError(error) {
return !!ignored.some((message) => error.message.includes(message));
}

/**
* @param {Buffer} data
*/
module.exports.fuzz = function (data) {
try {
const ast = Handlebars.parse(data.toString());
Handlebars.compile(ast, {});
} catch (error) {
if (error.message && !ignoredError(error)) {
throw error;
}
}
};
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#each goodbyes as |value index|}}{{index}}. {{value.text}}!{{/each}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#goodbyes}}{{this}}{{/goodbyes}}{{^goodbyes}}Right On!{{/goodbyes}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#each goodbyes}}{{@index}}. {{text}}! {{/each}}cruel {{world}}!
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#people}}\n{{name}}\n{{else if none}}\n{{none}}\n{{^}}\n{{/people}}\n
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#people}}{{name}}{{else if none}}{{none}}{{/people}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#people}}{{name}}{{else if none}}{{none}}{{else}}fail{{/people}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#goodbyes}}{{../name}}{{../name}}{{/goodbyes}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{foo.bar (baz bat=1)}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#with person as |foo|}}{{foo.first}} {{last}}{{/with}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#each goodbyes}}{{@index}}. {{text}}! {{#each ../goodbyes}}{{@index}} {{/each}}After {{@index}} {{/each}}{{@index}}cruel {{world}}!
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#each goodbyes as |value index|}}{{index}}. {{value.text}}! {{#each ../goodbyes as |childValue childIndex|}} {{index}} {{childIndex}}{{/each}} After {{index}} {{/each}}{{index}}cruel {{world}}!
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
\n {{#if}}\n{{/def}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#goodbyes}}{{text}}! {{/goodbyes}}cruel {{world}}!
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#outer}}Goodbye {{#inner}}cruel {{omg}}{{/inner}}{{/outer}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#helper}}{{*decorator}}{{/helper}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#goodbyes}}{{text}} cruel {{../name}}! {{/goodbyes}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#outer}}Goodbye {{#inner}}cruel {{../sibling}} {{../../omg}}{{/inner}}{{/outer}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#foo}} {{/foo}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#people}}\n{{name}}\n{{^}}\n{{none}}\n{{/people}}\n'
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{foo.bar baz "foo" true false bat=1}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#foo}} {{foo}}{{/foo}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#if goodbye includeZero=true}}GOODBYE {{/if}}cruel {{world}}!
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#helper}}{{*decorator foo}}{{/helper}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#goodbyes}}{{text}} cruel {{foo/../name}}! {{/goodbyes}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#helper}}{{#*decorator}}success{{/decorator}}{{/helper}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#none}}\n{{.}}\n{{^}}\nFail\n{{/none}}\n
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#people}}\n{{name}}\n{{else if none}}\n{{none}}\n{{/people}}\n'
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#each person}}{{#with .}}{{first}} {{last}}{{/with}}{{/each}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#people}}{{name}}{{^}}{{none}}{{/people}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#each data}}{{@index}}:{{a}} {{/each}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#data}}\n{{#if true}}\n{{.}}\n{{/if}}\n{{/data}}\nOK.
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#if goodbye}}GOODBYE {{/if}}cruel {{world}}!
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{foo}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#people}}{{name}}{{else if none}}{{none}}{{/if}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{foo.bar baz bat=1}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{*decorator foo}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#people}}{{name}}{{else if nothere}}fail{{else unless nothere}}{{none}}{{/people}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#goodbyes}}{{@index}}. {{text}}! {{/goodbyes}}cruel {{world}}!
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#each goodbyes}}{{#if @first}}{{text}}! {{/if}}{{/each}}cruel {{world}}!
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#helper}}{{#*decorator}}{{#*nested}}suc{{/nested}}cess{{/decorator}}{{/helper}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#with person}}{{first}} {{last}}{{/with}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#goodbyes}}{{text}}{{/goodbyes}} {{#goodbyes}}{{text}}{{/goodbyes}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#each goodbyes}}{{text}}! {{/each}}cruel {{world}}!
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#outer}}Goodbye {{#inner}}cruel {{omg.yes}}{{/inner}}{{/outer}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#with person}}Person is present{{else}}Person is not present{{/with}}'
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#each goodbyes}}{{@key}}. {{text}}! {{/each}}cruel {{world}}!
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
<b>#1</b>. goodbye! 2. GOODBYE! cruel world!
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#helper}}{{#*decorator}}suc{{/decorator}}{{#*decorator}}cess{{/decorator}}{{/helper}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
foo
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#none}}\n{{.}}\n{{^}}\n{{none}}\n{{/none}}\n
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#goodbyes}}{{/goodbyes}}cruel {{world}}!
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{foo.bar}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#with foo}}{{#if goodbye}}GOODBYE cruel {{../world}}!{{/if}}{{/with}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{*decorator "success"}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#people}}\n{{name}}\n{{^}}\n{{none}}\n{{/people}}\n
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{#helper}}{{*decorator}}suc{{/helper}}
18 changes: 18 additions & 0 deletions fuzz/test.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
#!/bin/bash

set -e

if [ $# -eq 0 ]; then
echo "Usage: $0 <max_total_time>
The <max_total_time> is passed to the internal fuzzing engine
(libFuzzer) to stop the fuzzing run after 'N' seconds."
exit 1
fi

max_total_time=$1

for i in ./*.fuzz.js; do
target=$(basename "$i" .js)
echo "-- Running $target for $max_total_time seconds."
npx jazzer $target corpus/$target --sync -- -max_total_time=$max_total_time;
done

0 comments on commit f1fffb4

Please sign in to comment.