-
Notifications
You must be signed in to change notification settings - Fork 3.5k
Description
Describe the issue
When using TreeEnsembleClassifier
with a LOGISTIC
post-transform, onnxruntime might report certain probabilities as zero, when the onnx reference evaluation would report them as very small, but non-zero probabilities (example is using python, but recreated with c++ as well)
# with LOGISTIC post-transform
onnxruntime: array([[1., 0.]], dtype=float32)
onnx reference evaluator: array([[9.9999994e-01, 3.0348513e-08]], dtype=float32)
If the post-transform is set to NONE
, and applying the logistic function manually, the results are closer together
# with NONE post-transform
onnxruntime: array([[ 17.310535, -17.310535]], dtype=float32)
onnx reference evaluator: array([[ 18.310518, -17.310518]], dtype=float32)
# applying sigmoid to both results manually produces
onnxruntime: array([[ 9.99999969652e-01, 3.0348005739e-08]], dtype=float32)
onnx reference evaluator: array([[ 9.99999988835e-01, -3.034852166e-08]], dtype=float32)
# which are much closer together
I believe that the ORT implementation of the logistic function is less precise than other implementations for large negative inputs.
To reproduce
Although the model I found this cannot be shared, this can be reproduced with any TreeEnsembleClassifier
that produces large (non-transformed) probabilities
Urgency
Without this fixed, we now need to disable post-transformation on our models, and applying the logistic post-process manually.
For future reference, the following snippet replaces a logistic post-transform with a sigmoid function
from onnx.helper import make_graph, make_model, make_node, make_tensor_value_info
from onnx import TensorProto, load_model, save_model
from onnx.compose import merge_graphs
def build_with_sigmoid(perm_onnx_path):
perm_model = load_model(perm_onnx_path)
perm_model.graph.node[0].attribute[-1].s = b"NONE"
probs_raw = make_tensor_value_info("probabilities_raw", TensorProto.FLOAT, [1, 2])
probs_trans = make_tensor_value_info("probabilities_transformed", TensorProto.FLOAT, [1, 2])
new_graph = make_graph([
make_node("Sigmoid", ["probabilities_raw"], ["probabilities_transformed"], name="Logistic")
], "Logistic", [probs_raw], [probs_trans])
new_model = make_model(
merge_graphs(perm_model.graph, new_graph, [("probabilities", "probabilities_raw")]),
opset_imports=perm_model.opset_import, ir_version=perm_model.ir_version
)
onnx_path = "model_sigmoid.onnx"
save_model(new_model, onnx_path)
Note that, for reasons I don't have time to get into, the sigmoid is also less precise than running numpy, but it's close enough for my purposes
Platform
Linux
OS Version
Ubuntu 22.04.5
ONNX Runtime Installation
Released Package
ONNX Runtime Version or Commit ID
1.23.1
ONNX Runtime API
Python
Architecture
X64
Execution Provider
Default CPU
Execution Provider Library Version
No response