0.5.0
Pre-release
Pre-release
Added cancellation token support (#38)
Note: While the updated method signature has a default value, this will break any extensions of the tokenization system as the TokenizeAsync
method has a new cancellation token parameter too.