[Bug]: out of bounds memory access

[notaz]

OpenVINO Version

master branch 05bf213

Operating System

Ubuntu 20.04 (LTS)

Device used for inference

CPU

Framework

None

Model used

No response

Issue description

There is out-of-bounds read in openvino when using a custom model. The issue can be reproduced with valgrind on master branch and many older releases too.

Step-by-step reproduction

Run the following program:

#include "openvino/runtime/core.hpp"
#include "openvino/op/ops.hpp"

static ov::Output<ov::Node> addInput(ov::ParameterVector &parameters)
{
	auto node = std::make_shared<ov::op::v0::Parameter>(
		ov::element::Type_t::f32, ov::PartialShape{1, 1, 240, 320});
	parameters.push_back(node);
	return node;
}

static ov::Output<ov::Node> addConv(ov::Output<ov::Node> &prevNode,
	int numOutput, const std::vector<int32_t> &pad,
	const std::vector<int32_t> &kernelSize, int group,
	const std::vector<size_t> &stride,
	const std::vector<size_t> &shape0,
	const std::vector<size_t> &shape1)
{
	std::vector<ptrdiff_t> padBegin = {pad[0], pad[0]};
	std::vector<ptrdiff_t> padEnd = {pad[0], pad[0]};
	ov::Output<ov::Node> output;
	if (pad.size() == 2)
	{
		padBegin = {pad[0], pad[1]};
		padEnd = {pad[0], pad[1]};
	}

	std::vector<size_t> dilation = {1, 1};

	size_t size = 1;
	for (size_t w : shape0)
		size *= w;
	std::vector<float> data(size);

	if (group == 1)
	{
		auto weightsConstant = std::make_shared<ov::op::v0::Constant>(
			ov::element::Type_t::f32, ov::Shape(shape0), data);

		auto convNode = std::make_shared<ov::op::v1::Convolution>(
			prevNode, weightsConstant, ov::Strides(stride),
			ov::CoordinateDiff(padBegin), ov::CoordinateDiff(padEnd),
			ov::Strides(dilation));
		output = convNode->output(0);
	}
	else
	{
		std::vector<size_t> shape;
		shape.push_back(group);
		shape.push_back(shape0[0] / group);
		for (size_t i = 1; i < shape0.size(); i++)
		{
			shape.push_back(shape0[i]);
		}

		auto weightsConstant = std::make_shared<ov::op::v0::Constant>(
			ov::element::Type_t::f32, ov::Shape(shape), data);

		auto convNode = std::make_shared<ov::op::v1::GroupConvolution>(
			prevNode, weightsConstant, ov::Strides(stride),
			ov::CoordinateDiff(padBegin), ov::CoordinateDiff(padEnd),
			ov::Strides(dilation));
		output = convNode->output(0);
	}

	size = 1;
	for (size_t w : shape1)
		size *= w;
	data.resize(size);
	data[0] = 1;

	std::vector<size_t> biasShape = {1, shape1[0], 1, 1};
	auto biasConstant = std::make_shared<ov::op::v0::Constant>(
		ov::element::Type_t::f32, ov::Shape(biasShape), data);

	auto biasNode = std::make_shared<ov::op::v1::Add>(output, biasConstant->output(0));
	return biasNode->output(0);
}

static ov::Output<ov::Node> addPrelu(ov::Output<ov::Node> &prevNode, int params)
{
	std::vector<float> data(params);
	auto slopeConstant = std::make_shared<ov::op::v0::Constant>(
		ov::element::Type_t::f32, ov::Shape({(size_t)params}), data);
	auto preluNode = std::make_shared<ov::op::v0::PRelu>(prevNode, slopeConstant);
	return preluNode->output(0);
}

int main()
{
	ov::Core core;
	ov::ParameterVector parameters{};
	ov::ResultVector results;
	ov::Output<ov::Node> output;

	output = addInput(parameters);
	output = addConv (output, 24, {1}, {3, 3}, 1, {1, 1}, {24, 1, 3, 3}, {24});
	output = addPrelu(output, 24);
	output = addConv (output, 24, {0, 0}, {3, 3}, 24, {2, 2}, {24, 1, 3, 3}, {24});
	output = addPrelu(output, 24);
	output = addConv (output, 40, {0}, {1}, 1, {1, 1}, {40, 24, 1, 1}, {40});
	output = addPrelu(output, 40);
	output = addConv (output, 40, {1, 1}, {3, 3}, 40, {1, 1}, {40, 1, 3, 3}, {40});
	output = addPrelu(output, 40);

	results.push_back(std::make_shared<ov::op::v0::Result>(output));
	std::shared_ptr<ov::Model> model = std::make_shared<ov::Model>(results, parameters);

	ov::CompiledModel cmodel = core.compile_model(model, "CPU");
	auto req = cmodel.create_infer_request();
	req.infer();
}

Relevant log output

==283798== Command: ./oob
==283798== 
==283798== Invalid read of size 8
==283798==    at 0xE17F01C: ???
==283798==    by 0x8CDCE59: ??? (in /tmp/openvino_master_rel/runtime/lib/intel64/libopenvino_intel_cpu_plugin.so)
...
==283798==  Address 0xcefb828 is 0 bytes after a block of size 8 alloc'd
==283798==    at 0x4846F95: operator new(unsigned long) (vg_replace_malloc.c:487)
==283798==    by 0x8A1A859: ??? (in /tmp/openvino_master_rel/runtime/lib/intel64/libopenvino_intel_cpu_plugin.so)

Issue submission checklist

• I'm reporting an issue. It's not a question.
• I checked the problem with the documentation, FAQ, open issues, Stack Overflow, etc., and have not found a solution.
• There is reproducer code and related data files such as images, videos, models, etc.

[rkazants]

Hi @notaz,

In your code, you start to infer without setting input data:

	auto req = cmodel.create_infer_request();
	req.infer();

Please check C++ samples to correct: https://github.com/openvinotoolkit/openvino/blob/master/samples/cpp/hello_classification/main.cpp#L108

Best regards,
Roman

[notaz]

I intentionally left out setting the input data because it has no effect on this bug and to minimize the reproducer. If I add

        std::vector<float> data(320*240);
        req.set_input_tensor({ov::element::f32, {1, 1, 240, 320}, data.data()});
        req.infer();

openvino still does the out-of-bounds read.

[rkazants]

I intentionally left out setting the input data because it has no effect on this bug and to minimize the reproducer. If I add

        std::vector<float> data(320*240);
        req.set_input_tensor({ov::element::f32, {1, 1, 240, 320}, data.data()});
        req.infer();

openvino still does the out-of-bounds read.

Try building your model from the top layers down to the lower layers, and at each step check the inference. This way, we can determine which layer is causing the error. It’s possible that the model is being constructed incorrectly for a certain layer.

Also, check that your environment works for inference of some other model, for example. sample.

Best regards,
Roman

[notaz]

This has already been done, if you remove the last layer in my reproducer the problem goes away.

Also, check that your environment works for inference of some other model, for example. sample.

Sample and other custom models work correctly without this issue.

[rkazants]

This has already been done, if you remove the last layer in my reproducer the problem goes away.

Could you please create a simple reproducer with this one last layer (single layer model)?

Best regards,
Roman

[notaz]

With only the last layer the problem goes away. The smallest I could get is 4 layers (attached), if more is removed the problem disappears. Perhaps it's related to some layer optimizations?

oob.cpp

[maxnick]

@notaz , thank you for the bug report and the reproducer. I'll look into this issue.