Skip to content

OpenVino seems to ignore the execution provider session #26291

@vlejeune-dxo

Description

@vlejeune-dxo

Describe the issue

Hi,

I'm using WindowsML + OpenVino provider on a Lunar Lake laptop,

It seems that the OpenVINO provider ignore the session options.

Here is a pseudo code for the workflow I'm implementing:

  auto env        = std::make_shared<Ort::Env>(ORT_LOGGING_LEVEL_VERBOSE, "OnnxInterface");
  winrt::Microsoft::Windows::AI::MachineLearning::ExecutionProviderCatalog catalog =
      winrt::Microsoft::Windows::AI::MachineLearning::ExecutionProviderCatalog::GetDefault();
  catalog.EnsureAndRegisterCertifiedAsync().get();

  // Populate execution provider devices
  std::unordered_map<std::string, std::vector<Ort::ConstEpDevice>> epDevices;
  for (const auto& ep : env->GetEpDevices())
  {
    DxO::FmtLogInfo("Available EP: {} - Device Name: {}", ep.EpName(), ep.Device().Vendor());
    epDevices[ep.EpName()].push_back(ep);
  }

  Ort::SessionOptions sessionOptions;
  sessionOptions.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_DISABLE_ALL);
  std::unordered_map<std::string, std::string> ep_options;
  ep_options["precision"]            = "FP32";
  ep_options["device_type"] = "GPU";
  sessionOptions.AppendExecutionProvider_V2(*m_env, {epDevices["OpenVINOExecutionProvider"]}, ep_options);

// Create session
auto session = std::make_unique<Ort::Session>(*env, modelInMemory.data(), modelInMemory.size(), sessionOptions);

  • Even if I put "GPU" as device_type, the NPU get selected. If I want to force the GPU, I have to filter the "epDevices["OpenVINOExecutionProvider"]" list
  • On a network with mixed FP16/FP32 precision, and if I filter the list to get only GPU, if I use the precision "ACCURACY" or "FP32", the result of the inference is wrong. If I only filter the list to get CPU, I get proper result.
    I'm expecting GPU FP32 and CPU to be close in term of result ; note that this network works properly with DirectML on the same hardware, and it also work with GPUs from other vendor with DirectML and their respective execution provider. I'm suspecting the provider is not honoring the "precision" mode and just use FP16 (which is know, for this network, to produce NaN values)

To reproduce

With a FP16 network,

  auto env        = std::make_shared<Ort::Env>(ORT_LOGGING_LEVEL_VERBOSE, "OnnxInterface");
  winrt::Microsoft::Windows::AI::MachineLearning::ExecutionProviderCatalog catalog =
      winrt::Microsoft::Windows::AI::MachineLearning::ExecutionProviderCatalog::GetDefault();
  catalog.EnsureAndRegisterCertifiedAsync().get();

  // Populate execution provider devices
  std::unordered_map<std::string, std::vector<Ort::ConstEpDevice>> epDevices;
  for (const auto& ep : env->GetEpDevices())
  {
    DxO::FmtLogInfo("Available EP: {} - Device Name: {}", ep.EpName(), ep.Device().Vendor());
    epDevices[ep.EpName()].push_back(ep);
  }

  Ort::SessionOptions sessionOptions;
  sessionOptions.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_DISABLE_ALL);
  std::unordered_map<std::string, std::string> ep_options;
  ep_options["precision"]            = "FP32";
  ep_options["device_type"] = "GPU";
  sessionOptions.AppendExecutionProvider_V2(*m_env, {epDevices["OpenVINOExecutionProvider"]}, ep_options);

// Create session
auto session = std::make_unique<Ort::Session>(*env, modelInMemory.data(), modelInMemory.size(), sessionOptions);

then bind+ run inference, and see if the result is close to the FP32 expected result or not, and if GPU was used or if it's the NPU.

Urgency

We manually select CPU for this network but the performance is obviously bad, we think it's an urgent issue

Platform

Windows

OS Version

26100.6584

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.23

ONNX Runtime API

WinML

Architecture

X64

Execution Provider

OpenVINO

Execution Provider Library Version

1.8.15.0 (KB5067990)

Metadata

Metadata

Assignees

No one assigned

    Labels

    ep:OpenVINOissues related to OpenVINO execution provider

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions