🚧 Work in progress.
Use humaneval.py --template to apply templates to humaneval interview suite turning them into prompts suitable for interview.
Note the format here is different, uses a single {{prompt}} variable as the input.
Vicuna-1p1-HumanEval.txt
Wizard-7B-HumanEval.txt
Llama-Humaneval.txt
Use humaneval.py --answers to convert any interview output into .jsonl format and use the upsteam evaluator: https://github.com/openai/human-eval