❓Get stats (e.g. counts) about the merged pairs

[pietrolesci]

Hi there,

I was wondering whether there is an easy way to ask the tokeniser trainer to return the counts (or frequency) of the pair in the moment the merge decision is made. The ideal output would be having a new column with the frequency of the pair in the merges.txt file.

Tagging the line below as it seems the relevant part of the code that has the info (i.e., pair_counts)

tokenizers/tokenizers/src/models/bpe/trainer.rs

Line 465 in 25aee8b

let (mut pair_counts, mut where_to_update) = self.count_pairs(&words, &counts, &progress);

Thanks a lot for your help!

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[ArthurZucker]

Would something like a log do the trick for you @pietrolesci ? Cuz I plan on adding some log helper to help non rust users debug the internals