.Net: Examples encourage stateless token counter functions

[tonybaloney]

Describe the bug

The sample functions in Example55_textchunker.cs show how to use a BPE encoding function to replace the builtin left-shift approximation function:

https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/KernelSyntaxExamples/Example55_TextChunker.cs#L87C5-L108

These examples encourage a stateless function that instantiates the BPE for each call. Both for sharp token and the new ML.Tokenizers API, this is really inefficient. Instead, we should encourage usage with a class that stores the tokenizer and reuses the count tokens method as the TokenCount delegate:

public class StatefulTokenCounter
    {
        private readonly Tokenizer _tokenizer;

        public StatefulTokenCounter(string model)
        {
            this._tokenizer = Tiktoken.CreateByModelNameAsync(model).Result;
        }
        public int Count(string input)
        {
            return this._tokenizer.CountTokens(input);
        }
    }


var counter = new StatefulTokenCounter("gpt-4");
var lines = TextChunker.SplitPlainTextLines("This is a test", 40, counter.Count);

I tested side-by-side with a stateful class passing the instance method to the delegate. For a small input, there was a 5x difference in performance, trailing to a 2x for a large input.