Skip to content

add tokenize_document from pie-modules #462

Open
@ArneBinder

Description

@ArneBinder

tokenize_document is PyTorch specific (it uses a transformers tokenizer), so it will be removed from pie-modules (see ArneBinder/pie-modules#187).

more context: ArneBinder/pie-core#17

requires pie-modules release including ArneBinder/pie-modules#187

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions