[BUG] Truncating Line by Characters not Tokens

[YouNeedCryDear]

Regarding PR here #20

Obviously we are truncating the line by number of tokens instead of using characters.
This is OK for small dataset like Sonnet, with few words for each line. But if data contains only one line or lots of words in a line, the discrepancy will be significant.

@CatherineSue Could you elaborate on why the dependency on calculating the char to token ration for the data in sampler is not wanted? We do calculate the char to token ratio for sonnet dataset anyway at the beginning of the application. Should we just pass the char to token ratio for sonnet dataset into this function?

[YouNeedCryDear]

Reference of how vLLM process sonnet dataset.
https://github.com/vllm-project/vllm/blob/main/benchmarks/benchmark_dataset.py#L540
Average number of tokens per line is calculated during initiation and then sampling is based on token level.