Add new nvtext::normalize_characters API

[davidwendt]

Description

Adds new normalizer APIs as part of the rework for the subword-tokenizer.
The new API is split into 2 parts. First a normalizer object is created with appropriate state: lower-case and special-tokens. The normalizing tables are currently hardcoded inside libcudf. Future versions of the this may load these tables from some other source. The 2nd API is given the input strings column and the normalizer object and returns a normalized strings column. The normalizer object can be reused on all subsequent normalize_characters calls.

The current nvtext::normalize_characters loads the normalizing tables on each call which can be significant overhead. This API will be deprecated and replaced by these 2 new ones. Some utility functions from that implementation have been refactored to be used by both until the old one is removed.

The first API creates the normalizer object.

std::unique_ptr<character_normalizer> create_character_normalizer(
  bool do_lower_case,
  cudf::strings_column_view const& special_tokens,
  rmm::cuda_stream_view stream,
  rmm::device_async_resource_ref mr);

The 2nd API uses the normalizer on a strings column:

std::unique_ptr<cudf::column> normalize_characters(
  cudf::strings_column_view const& input,
  character_normalizer const& normalizer,
  rmm::cuda_stream_view stream,
  rmm::device_async_resource_ref mr);

Using the python interface:

import cudf
from cudf.core.character_normalizer import CharacterNormalizer

cn = CharacterNormalizer(do_lower=False)
sn = cn.normalize(input_strings)

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[davidwendt]

/ok to test

[davidwendt]

/ok to test

[GregoryKimball]

@kingcrimsontianyu and @karthikeyann would you two please lead the review for this contribution?

[kingcrimsontianyu]

Would it be better to use std::unique_ptr<character_normalizer_impl> for the impl data member?

[kingcrimsontianyu]

Looked it up in the dictionary as I thought this is a literature word I didn't know 😃

[davidwendt]

I don't really know what this is but tests the normalizer paths I want. Hopefully it is not something offensive.

[kingcrimsontianyu]

Just a question: Is this because some iterators in libcudf or user-defined iterators may (incorrectly) use int as the iterator's difference_type as opposed to std::ptrdiff_t? I'm curious when this is usually happening.

[davidwendt]

Many of the thrust functions I have 2 different code paths depending on the range of the input iterators -- 32-bit and 64-bit. Unfortunately, building with thrust pulls in both even if we only use 1 of them. This bloats the code upto 2x in some cases. We generally only require 32-bit (size_type) iterator ranges and we are able to compile out the thrust 64-bit iterator paths with some strategic patching in our cmake -- significantly reducing our binary size. Note, the bloat issue is something the CCCL team is working on so the patch we use is meant to be temporary.

Anyway, this means in places were we actually a need 64-bit iterator range (like for reading large strings character vectors we currently support), we need to workaround the missing 64-bit range thrust APIs. The remove_copy_safe function and the remove_safe utility below workaround the limit by calling the underlying thrust functions in batches of max<int> counts.