Initial fuzz integration

.github/workflows/ci.yml

@@ -58,6 +58,10 @@ jobs:
           cd ./tests/integration/webpack-babel-test && ./test.sh && cd -
           cd ./tests/integration/webpack-test && ./test.sh && cd -
 
+      - name: Fuzz Test
+        # Runs for 2 minutes
+        run: cd ./fuzz && ./test.sh 120 && cd -
+
   browser:
     name: Test (Browser)
     runs-on: 'ubuntu-22.04'

fuzz/README.md

@@ -0,0 +1,45 @@
+# Fuzz Testing
+
+Fuzz testing is:
+
+> An automated software testing technique that involves providing invalid, unexpected, or random data as inputs to a program.
+
+We use coverage guided fuzz testing to automatically discover bugs in Handlebars.js.
+
+This `fuzz/` directory contains the configuration and the fuzz tests for Handlebars.js.
+To generate and run fuzz tests, we use the [Jazzer.js](https://github.com/CodeIntelligenceTesting/jazzer.js/) library.
+
+## Running a fuzzer
+
+This directory contains fuzzers like for example `compiler.fuzz`. You can run it with:
+
+```sh
+$ npx jazzer fuzz/compiler.fuzz fuzz/corpus/compiler.fuzz/ --sync
+```
+
+You should see output that looks something like this:
+
+```
+#7      INITED cov: 20 ft: 20 corp: 1/4b exec/s: 0 rss: 162Mb
+#20     REDUCE cov: 20 ft: 20 corp: 1/3b lim: 4 exec/s: 0 rss: 162Mb L: 3/3 MS: 3 CopyPart-ChangeBit-EraseBytes-
+#45     REDUCE cov: 20 ft: 20 corp: 1/2b lim: 4 exec/s: 0 rss: 162Mb L: 2/2 MS: 5 CrossOver-ChangeByte-CopyPart-CopyPart-EraseBytes-
+#46     REDUCE cov: 20 ft: 20 corp: 1/1b lim: 4 exec/s: 0 rss: 162Mb L: 1/1 MS: 1 EraseBytes-
+#219    REDUCE cov: 25 ft: 25 corp: 2/4b lim: 4 exec/s: 0 rss: 162Mb L: 3/3 MS: 3 ChangeBit-ChangeBit-CMP- DE: "\001\000"-
+#220    REDUCE cov: 25 ft: 25 corp: 2/3b lim: 4 exec/s: 0 rss: 162Mb L: 2/2 MS: 1 EraseBytes-
+#293    REDUCE cov: 25 ft: 25 corp: 2/2b lim: 4 exec/s: 0 rss: 162Mb L: 1/1 MS: 3 ChangeByte-ShuffleBytes-EraseBytes-
+```
+
+It will continue to generate random inputs forever, until it finds a
+bug or is terminated. The testcases for bugs it finds can be seen in
+the form of `crash-*` or `timeout-*` at the place from where command is run.
+You can rerun the fuzzer on a single input by passing it on the
+command line `npx jazzer fuzz/compiler.fuzz /path/to/testcase`.
+
+## The corpus
+
+The corpus is a set of test inputs, stored as individual files,
+provided to the fuzz target as a starting point (to “seed” the mutations).
+The quality of the seed corpus has a huge impact on fuzzing efficiency;
+the higher the quality, the easier it is for the fuzzer to discover new code paths.
+The ideal corpus is a minimal set of inputs that provides maximal code coverage
+Each fuzzer has an individual corpus under `fuzz/corpus/test_name`.

fuzz/compiler.fuzz.js

@@ -0,0 +1,27 @@
+const Handlebars = require('../dist/cjs/handlebars')['default'];
+
+const ignored = [
+  'Parse error',
+  'Lexical error',
+  `doesn't match`,
+  'Invalid path',
+  'Unsupported number of partial arguments',
+];
+
+function ignoredError(error) {
+  return !!ignored.some((message) => error.message.includes(message));
+}
+
+/**
+ * @param {Buffer} data
+ */
+module.exports.fuzz = function (data) {
+  try {
+    const ast = Handlebars.parse(data.toString());
+    Handlebars.compile(ast, {});
+  } catch (error) {
+    if (error.message && !ignoredError(error)) {
+      throw error;
+    }
+  }
+};

fuzz/corpus/compiler.fuzz/049e4aa6076c8e4c87f9d10c0194228c6fdfd8b6

@@ -0,0 +1 @@
+{{#each goodbyes as |value index|}}{{index}}. {{value.text}}!{{/each}}

fuzz/corpus/compiler.fuzz/10d14556dcdc478fe0df984e0743bb72634f92a6

@@ -0,0 +1 @@
+{{#goodbyes}}{{this}}{{/goodbyes}}{{^goodbyes}}Right On!{{/goodbyes}}

fuzz/corpus/compiler.fuzz/177d2dae884490b937c7c004b750c45bba7d731b

@@ -0,0 +1 @@
+{{#each goodbyes}}{{@index}}. {{text}}! {{/each}}cruel {{world}}!

fuzz/corpus/compiler.fuzz/1e6bfe29056662482a7fc7e724ab1139fd8f087f

@@ -0,0 +1 @@
+{{#people}}\n{{name}}\n{{else if none}}\n{{none}}\n{{^}}\n{{/people}}\n

fuzz/corpus/compiler.fuzz/1f0ef523dad6722ca31003812b64347679b430e3

@@ -0,0 +1 @@
+{{#people}}{{name}}{{else if none}}{{none}}{{/people}}

fuzz/corpus/compiler.fuzz/220dde18dfb07f093dc5a64be5b71797264176b3

@@ -0,0 +1 @@
+{{#people}}{{name}}{{else if none}}{{none}}{{else}}fail{{/people}}

fuzz/corpus/compiler.fuzz/24a7c760429e57cb3a204aebf6124b83a91c21b9

@@ -0,0 +1 @@
+{{#goodbyes}}{{../name}}{{../name}}{{/goodbyes}}

fuzz/corpus/compiler.fuzz/325ad48a5aaa0a33f2d122a1f1e5012b882f8eef

@@ -0,0 +1 @@
+{{foo.bar (baz bat=1)}}

fuzz/corpus/compiler.fuzz/345411d48fafb2941b540e7465e260e9fd108825

@@ -0,0 +1 @@
+{{#with person as |foo|}}{{foo.first}} {{last}}{{/with}}

fuzz/corpus/compiler.fuzz/359a4cd9b9cb6fc0539da3a89d3a6b169048f419

@@ -0,0 +1 @@
+{{#each goodbyes}}{{@index}}. {{text}}! {{#each ../goodbyes}}{{@index}} {{/each}}After {{@index}} {{/each}}{{@index}}cruel {{world}}!

fuzz/corpus/compiler.fuzz/377ccad52355aeec2799cd6fe1814eb0d35ecbb0

@@ -0,0 +1 @@
+{{#each goodbyes as |value index|}}{{index}}. {{value.text}}! {{#each ../goodbyes as |childValue childIndex|}} {{index}} {{childIndex}}{{/each}} After {{index}} {{/each}}{{index}}cruel {{world}}!

fuzz/corpus/compiler.fuzz/479c14fbc04f9b507295d766983410f69cb33651

@@ -0,0 +1 @@
+ \n  {{#if}}\n{{/def}}

fuzz/corpus/compiler.fuzz/541ee4fffafb12da4a8ad4c35d3b74f837fa666b

@@ -0,0 +1 @@
+{{#goodbyes}}{{text}}! {{/goodbyes}}cruel {{world}}!

fuzz/corpus/compiler.fuzz/571ed3dcd0160af551c641afb7fdc77263706894

@@ -0,0 +1 @@
+{{#outer}}Goodbye {{#inner}}cruel {{omg}}{{/inner}}{{/outer}}

fuzz/corpus/compiler.fuzz/68b88717c0b2df6a4a3de21c4d769d451c682fa7

@@ -0,0 +1 @@
+{{#helper}}{{*decorator}}{{/helper}}

fuzz/corpus/compiler.fuzz/6bfc672351bbb03a31a585bc2a6f2c39659ed6ea

@@ -0,0 +1 @@
+{{#goodbyes}}{{text}} cruel {{../name}}! {{/goodbyes}}

fuzz/corpus/compiler.fuzz/75be635b36860c89b89925071b3be70eaadc620d

@@ -0,0 +1 @@
+{{#outer}}Goodbye {{#inner}}cruel {{../sibling}} {{../../omg}}{{/inner}}{{/outer}}

fuzz/corpus/compiler.fuzz/7778d3ba28e00fe7932c0b76072d47e808f2af75

@@ -0,0 +1 @@
+{{#foo}} {{/foo}}

fuzz/corpus/compiler.fuzz/77c7281e90c74b6d24fa8354bf5a4af5da04b175

@@ -0,0 +1 @@
+{{#people}}\n{{name}}\n{{^}}\n{{none}}\n{{/people}}\n'

fuzz/corpus/compiler.fuzz/788d1b5577729daba2f4bc759f7088e93b8a9d75

@@ -0,0 +1 @@
+{{foo.bar baz "foo" true false bat=1}}

fuzz/corpus/compiler.fuzz/8ab76bb91bc93adef82538cee689f5182e50b770

@@ -0,0 +1 @@
+{{#foo}} {{foo}}{{/foo}}

fuzz/corpus/compiler.fuzz/9039e59adf416f481d16e698e97e5224a62e09be

@@ -0,0 +1 @@
+{{#if goodbye includeZero=true}}GOODBYE {{/if}}cruel {{world}}!

fuzz/corpus/compiler.fuzz/9b057b8346de564353fdfb28f93d2bc8eeb25b9d

@@ -0,0 +1 @@
+{{#helper}}{{*decorator foo}}{{/helper}}

fuzz/corpus/compiler.fuzz/9bd641c7d52363fd042a59e9ff2ea5efa4e6bdd6

@@ -0,0 +1 @@
+{{#goodbyes}}{{text}} cruel {{foo/../name}}! {{/goodbyes}}

fuzz/corpus/compiler.fuzz/9c973dfdc395aa5a7b61930555454baa2c040d30

@@ -0,0 +1 @@
+{{#helper}}{{#*decorator}}success{{/decorator}}{{/helper}}

fuzz/corpus/compiler.fuzz/a4210f0ffc7681710c774caed72c55c0a8b2dcb5

@@ -0,0 +1 @@
+{{#none}}\n{{.}}\n{{^}}\nFail\n{{/none}}\n

fuzz/corpus/compiler.fuzz/a54780eb3853807872501547fa05816821a8ce65

@@ -0,0 +1 @@
+{{#people}}\n{{name}}\n{{else if none}}\n{{none}}\n{{/people}}\n'

fuzz/corpus/compiler.fuzz/a612a6be8064d1c23c1f1bd6044080b0b3d696b7

@@ -0,0 +1 @@
+{{#each person}}{{#with .}}{{first}} {{last}}{{/with}}{{/each}}

fuzz/corpus/compiler.fuzz/aea23e0178cca7f24c5154677191f4052aedef89

@@ -0,0 +1 @@
+{{#people}}{{name}}{{^}}{{none}}{{/people}}

fuzz/corpus/compiler.fuzz/b1725265f4a3d72f5b14025cbe505eae3ab952e5

@@ -0,0 +1 @@
+{{#each data}}{{@index}}:{{a}} {{/each}}

fuzz/corpus/compiler.fuzz/b8667a687aa3152044123f9b7edc48c2df35d17b

@@ -0,0 +1 @@
+{{#data}}\n{{#if true}}\n{{.}}\n{{/if}}\n{{/data}}\nOK.

fuzz/corpus/compiler.fuzz/b943e7c833f314d19dfeebacc0d9cce72353f838

@@ -0,0 +1 @@
+{{#if goodbye}}GOODBYE {{/if}}cruel {{world}}!

fuzz/corpus/compiler.fuzz/c428bff0953db1418224d111cb6aa4acffbb6e80

@@ -0,0 +1 @@
+{{foo}}

fuzz/corpus/compiler.fuzz/c7a41bfe4d495b72015c1a27378e4eca2ab43974

@@ -0,0 +1 @@
+{{#people}}{{name}}{{else if none}}{{none}}{{/if}}

fuzz/corpus/compiler.fuzz/cc11925603f5dac917d2ee315a651ad66b048406

@@ -0,0 +1 @@
+{{foo.bar baz bat=1}}

fuzz/corpus/compiler.fuzz/cf0f1476c57ef0df5e436777250049aa52c889c2

@@ -0,0 +1 @@
+{{*decorator foo}}

fuzz/corpus/compiler.fuzz/d33e71cc5dd85a36dc84c8bf12dabc0624e3967d

@@ -0,0 +1 @@
+{{#people}}{{name}}{{else if nothere}}fail{{else unless nothere}}{{none}}{{/people}}

fuzz/corpus/compiler.fuzz/d3ae5d5af973778c7b0cd938bd75cee20c5f0037

@@ -0,0 +1 @@
+{{#goodbyes}}{{@index}}. {{text}}! {{/goodbyes}}cruel {{world}}!

fuzz/corpus/compiler.fuzz/dbc7b50baec9f9d29021604856f56bc73f378a5b

@@ -0,0 +1 @@
+{{#each goodbyes}}{{#if @first}}{{text}}! {{/if}}{{/each}}cruel {{world}}!

fuzz/corpus/compiler.fuzz/dccbe9317b88449ef1e3b7b82060dfd89fcf1bc2

@@ -0,0 +1 @@
+{{#helper}}{{#*decorator}}{{#*nested}}suc{{/nested}}cess{{/decorator}}{{/helper}}

fuzz/corpus/compiler.fuzz/ddd1b10efb1b50e68c0172d5d5a0da88adc24604

@@ -0,0 +1 @@
+{{#with person}}{{first}} {{last}}{{/with}}

fuzz/corpus/compiler.fuzz/df2c4e4bd47373f5365e314084a82d08846d6568

@@ -0,0 +1 @@
+{{#goodbyes}}{{text}}{{/goodbyes}} {{#goodbyes}}{{text}}{{/goodbyes}}

fuzz/corpus/compiler.fuzz/e2a0e338fc07eaf160bfb0977e398b7627bd6c21

@@ -0,0 +1 @@
+{{#each goodbyes}}{{text}}! {{/each}}cruel {{world}}!

fuzz/corpus/compiler.fuzz/e35b817b11839df03d4da6e1d93d829fad305788

@@ -0,0 +1 @@
+{{#outer}}Goodbye {{#inner}}cruel {{omg.yes}}{{/inner}}{{/outer}}

fuzz/corpus/compiler.fuzz/e3d4807bde97804672139e5a0b194be4943263a2

@@ -0,0 +1 @@
+{{#with person}}Person is present{{else}}Person is not present{{/with}}'

fuzz/corpus/compiler.fuzz/e42f86f8128679fc2edadb3579fbbfd3c4a7e5dd

@@ -0,0 +1 @@
+{{#each goodbyes}}{{@key}}. {{text}}! {{/each}}cruel {{world}}!

fuzz/corpus/compiler.fuzz/e73f9110dae281e494e6df91ac56f86c5116b2e2

@@ -0,0 +1 @@
+<b>#1</b>. goodbye! 2. GOODBYE! cruel world!

fuzz/corpus/compiler.fuzz/e788c92d3e4eba1571d6c89345be815064363eae

@@ -0,0 +1 @@
+{{#helper}}{{#*decorator}}suc{{/decorator}}{{#*decorator}}cess{{/decorator}}{{/helper}}

fuzz/corpus/compiler.fuzz/f1d2d2f924e986ac86fdf7b36c94bcdf32beec15

@@ -0,0 +1 @@
+foo

fuzz/corpus/compiler.fuzz/f3c05d4e447d686faa78e8784a4b528b4ec044c3

@@ -0,0 +1 @@
+{{#none}}\n{{.}}\n{{^}}\n{{none}}\n{{/none}}\n

fuzz/corpus/compiler.fuzz/f422726a28d4ee1c785cb66c2233ce4e5892bdfb

@@ -0,0 +1 @@
+{{#goodbyes}}{{/goodbyes}}cruel {{world}}!

fuzz/corpus/compiler.fuzz/f6cd65165fede63d9c262cc983b4bfd28e91a7b8

@@ -0,0 +1 @@
+{{foo.bar}}

fuzz/corpus/compiler.fuzz/fb7eadc1d17bffee94eac2c2f7160ea58615304b

@@ -0,0 +1 @@
+{{#with foo}}{{#if goodbye}}GOODBYE cruel {{../world}}!{{/if}}{{/with}}

fuzz/corpus/compiler.fuzz/fc105bb181fd4b89c4c70db143d6736d02b1a96c

@@ -0,0 +1 @@
+{{*decorator "success"}}

fuzz/corpus/compiler.fuzz/fe6f8d8f27df2ede214191fd0baeb3e7efb9c039

@@ -0,0 +1 @@
+{{#people}}\n{{name}}\n{{^}}\n{{none}}\n{{/people}}\n

fuzz/corpus/compiler.fuzz/fed10ca600aaee904b9bd07e6f438324fdf23fad

@@ -0,0 +1 @@
+{{#helper}}{{*decorator}}suc{{/helper}}

fuzz/test.sh

@@ -0,0 +1,18 @@
+#!/bin/bash
+
+set -e
+
+if [ $# -eq 0 ]; then
+  echo "Usage: $0 <max_total_time> 
+The <max_total_time> is passed to the internal fuzzing engine 
+(libFuzzer) to stop the fuzzing run after 'N' seconds."
+  exit 1
+fi
+
+max_total_time=$1
+
+for i in ./*.fuzz.js; do
+  target=$(basename "$i" .js)
+  echo "-- Running $target for $max_total_time seconds."
+  npx jazzer $target corpus/$target --sync -- -max_total_time=$max_total_time;
+done