Open
Description
private final String encodingExampleWithQuotes = "Hello my name is "Kevin".";
When testing with a string that has quotes I get error.
GPT2Tokenizer tokenizer = GPT2Tokenizer.fromPretrained("tokenizers/gpt2");
List<Integer> result = tokenizer.encode(encodingExampleWithQuotes);
Assertions.assertEquals(decodingExampleWithQutes, result);
java.lang.NullPointerException: Cannot invoke "java.math.BigInteger.intValue()" because the return value of "java.util.Map.get(Object)" is null
Metadata
Metadata
Assignees
Labels
No labels