io-engine Resource configuration in a cluster

[whg517]

I am using openebs to provide volume storage for my production k8s cluster. I am currently using three machines with 10G fiber networks and nvme disks as io-engine nodes.

I just started with the default resource configuration at helm chart and the io-engine had a CPU of 2. I can see from monitoring today that its CPU usage is 99%. I have now adjusted the CPU to 8 , and when the adjustment is complete, the CPU of the three pods is still 99%.

I would like to ask whether all cpus are used by default for io-engine or the CPU I currently allocate is still insufficient?

[tiagolobocastro]

The io-engine is constantly polling for IO so the full CPU usage is currently expected.
There's a ticker here to allow for interrupt based.

[whg517]

@tiagolobocastro Hello, according to your answer, my understanding is that it is normal for the CPU of io-enging to be at 99%, right? So no matter how many cpus I adjust to, it's going to be 99%?

How can I tell if the io-engine has a load bottleneck? Do I need to add CPU resources to it where appropriate, or will the resources it is currently using become a bottleneck for the entire pod usage volume?

Is there any way to tell or guide me to make appropriate resource tuning based on my cluster load for future use of openebs?

The k8s cluster I am currently building in the IDC room has reached 7 nodes and is gradually migrating the workload, now using three 10G cable networks with two 1.7T PCI4*4 NVME disks each. We continue to add nodes to the cluster. I would like to see more stable openebs provide the underlying storage for our cluster.

[tiagolobocastro]

@tiagolobocastro Hello, according to your answer, my understanding is that it is normal for the CPU of io-enging to be at 99%, right? So no matter how many cpus I adjust to, it's going to be 99%?

Yes that's correct.

How can I tell if the io-engine has a load bottleneck? Do I need to add CPU resources to it where appropriate, or will the resources it is currently using become a bottleneck for the entire pod usage volume?

hmm we might need to start exposing some metrics for this such as nvme-target queue length for example. cc @avishnu @dsharma-dc

Is there any way to tell or guide me to make appropriate resource tuning based on my cluster load for future use of openebs?

Not as of today, but this definitely would be useful to add.

The k8s cluster I am currently building in the IDC room has reached 7 nodes and is gradually migrating the workload, now using three 10G cable networks with two 1.7T PCI4*4 NVME disks each. We continue to add nodes to the cluster. I would like to see more stable openebs provide the underlying storage for our cluster.

This is all good feedback thank you. Would you be able to raise github feature requests?

[whg517]

@tiagolobocastro

Thank you for your reply.

Based on the documentation Performance Tips, I found that CPU isolation can be used to optimize performance. I will try to optimize after adding io-engine nodes and continue to focus on CPU usage.

Since I don't have a clear understanding of the inner workings of openebs, I don't have the ability to contribute to the community yet, and I may try to contribute features as I get deeper into it. Thank you for your trust.

Now that I understand the cause of the problem, I will mark it as close. Thank you for your help.