Approximate tokens len

[MichaReiser]

Summary

The parser benchmark has become flaky after merging #9466.
A possible reason for the flakiness could be some non-determinism in jemalloc that makes it execute different branches while collecting the tokens into a Vec.

The idea of this PR is to approximate the size of the tokens Vec based on the source code length. The approximation used here is based on an analysis of the CPython code base and our ecosystem projects (including aiflow, black, django, transformers, twine, warehouse, zulip). I added a console output that prints the source code length and resulting tokens vec size for each file and piped it into a CSV. I then used a small python script to paint the $buffer/source$ distribution and picked 0.15 as a lower bound for the size of the tokens vec.

Ecosystem

CPython

Note: The 1 looks suspicious. It is due to empty files. I'm not sure why there are so many empty files

Source

Test Plan

I expected some real world perf improvements from this work but neither our hyperfine benchmarks nor the parser's microbenchmarks show any real improvement 🤷

[github-actions]

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.

[charliermarsh]

Neat idea.