Tokenize correctly in languages that don't demarcate with whitespace

[stenskjaer]

Examples of such languages are Sanskrit and Arabic.

There an app note can contain only a fragment of a word...