Return source white space string

[kino-ma]

Currently, Node.white_space always returns half-width space " " (0x20).

However, it is not desirable behavior in my use case because I want to preserve source white space characters other than " " (e.g., "\t") also.

By this PR, the function will return the original white space string.

[polm]

Thank you, this looks like an excellent change. Thank you also for including tests.

It may take me a little bit to review it, but I'll check it over and get it included.

[kino-ma]

Thanks for your reply!
よろしくお願いいたします 🙏

[polm]

Sorry it took me a while to get to this. The basic code is good, but because of the way MeCab does memory management, it has the issue that the surface is invalidated by further calls to parseToNode. For example:

from fugashi import Tagger

tagger = Tagger()

x = tagger("a\nb")
print(repr(x[1].white_space))
# => '\n'
y = tagger("a\tb")
print(repr(x[1].white_space))
# => '\t' (this is wrong)

The solution is to cache the white space, as is done for the token surfaces. I can add this myself.

[polm]

OK, caching is implemented so this should actually work now. I will do a few more checks before merging. Might take a little while, though it shouldn't be another three weeks 😅

[kino-ma]

Thank you so much!! 😆