can't load a tokneizer from json file, the error is "invaild argument"

[YuzheZhou2000]

"Can't load a tokenizer from the JSON file, the error is 'invalid argument'. This issue occurs randomly, and I can assure that the JSON file is correct. Does anyone know what might be causing this problem?

[daulet]

Randomly as in non-deterministically? Can you share a minimal repro?

[usepr]

Don't know why err was set to EINVAL after C.from_file() or C.from_bytes() in some specific environments, even though the tokenizer returns correctly. We can check the return value of C.from_file(), not err.

func FromFile(path string) (*Tokenizer, error) {
	cPath := C.CString(path)
	defer C.free(unsafe.Pointer(cPath))
-	tokenizer, err := C.from_file(cPath)
-	if err != nil {
-		return nil, err
-	}
+	tokenizer := C.from_file(cPath)
+	if tokenizer == nil {
+		return nil, error.New("blah blah")
+	}
	return &Tokenizer{tokenizer: tokenizer}, nil
}