OpenTelemetry Collector Memory Issues

[andrewgkew]

I have opentelemery collector setup to scrape 4 endpoints

1. skywalking metrics
2. kubernetes cadvisor
3. kube-state-metrics
4. stiod
5. envoy-stats

I have noticed that the data coming into SkyWalking has been really slow, so I started to investigate and what I found is that the otel collector pod keeps crashing with status OOMKilled status, which is an out of memory issue.

So I figured either there is too much for it to collect or it needs more memory

Looking into SkyWalking k8s monitoring I cant see this as a problem there are all so I tried a couple of things

1. Increase the replicas - I thought more pods to process the data should help
   OUTCOME: They all now crash with the same OOMKilled status

2. Increase their memory request - I increase this from 400Mi to 800Mi
   OUTCOME: Nothing changed there they all still crash with OOMKilled

Checking the node resources using k top nodes at the time the pods are crashing its at approx 60% so it has not over shot its memory on the actual machine

Question: Have you seen this issue before and/or any advice on how to handle this?

Do I have the collector setup with too many jobs and maybe should separate them?

Thanks

[wu-sheng]

I think you should raise this question to OpenTelemetry community. We don't use any specific feature or change any OpenTelemetry codes. So, I would guess this is some kind of bugs of the version you are using.

[andrewgkew]

ok thanks @wu-sheng - I guess it was more if you had come across the issue before.

I had an issue getting it to work in the beginning and @wankai123 suggested I use another version of the collector as you have experienced an issue with the version defined in your docs.

If this is not something you have come across I will pose the question on their boards

Thanks

[andrewgkew]

Anyone who sees this in the future I have created a question for the OTEL community

open-telemetry/opentelemetry-collector#4010

[wu-sheng]

OpenTelemetry collector has issues(blocking, OOM from time to time) in the past, we suffered, and aren't sure why they keep happening.
OpenTelemetry in the CNCF, they run projects based on different community model, which we can't judge.

We also find out this is out of control, so, you can see this in the dev version.
https://skywalking.apache.org/docs/skywalking-satellite/latest/en/setup/plugins/fetcher_prometheus-metrics-fetcher/

But at the same time, I wouldn't recommend you to use a dev version of Satellite, and also it doesn't have backend MAL setting yet.

[andrewgkew]

Ok thanks @wu-sheng - luckily right now this is just for my conference demo so I can get by with the issue for now, will see what the OTEL community say too and go from there.

[andrewgkew]

@wu-sheng they have come up with some interesting suggestions, two I think might help

1. Setting up the collector to be a daemonset and then configuring the jobs per node
2. Dropping metrics from cadvisor and KSM that arent used - are you able to help with which metrics SkyWalking uses so I can try and drop metrics?

I think that 1. might be the quickest option for now and maybe 2 for a longer term solution

[wankai123]

I think OAP doesn't use OTEL to receive the envoy-stats, it uses gRPC directly, and envoy-stats has too many metrics, you should filter them otherwise the collector is very easy to crash.
eg in doc: https://skywalking.apache.org/docs/main/latest/en/setup/envoy/metrics_service_setting/#send-envoy-metrics-to-skywalking-with--without-istio:

istioctl manifest install -y \
  --set profile=demo # replace the profile as per your need \
  --set meshConfig.defaultConfig.envoyMetricsService.address=<skywalking.address.port.11800> \ # replace <skywalking.address.port.11800> with your actual SkyWalking OAP address
  --set 'meshConfig.defaultConfig.proxyStatsMatcher.inclusionRegexps[0]=.*membership_healthy.*' \
  --set 'meshConfig.defaultConfig.proxyStatsMatcher.inclusionRegexps[1]=.*upstream_cx_active.*' \
  --set 'meshConfig.defaultConfig.proxyStatsMatcher.inclusionRegexps[2]=.*upstream_cx_total.*' \
  --set 'meshConfig.defaultConfig.proxyStatsMatcher.inclusionRegexps[3]=.*upstream_rq_active.*' \
  --set 'meshConfig.defaultConfig.proxyStatsMatcher.inclusionRegexps[4]=.*upstream_rq_total.*' \
  --set 'meshConfig.defaultConfig.proxyStatsMatcher.inclusionRegexps[5]=.*upstream_rq_pending_active.*' \
  --set 'meshConfig.defaultConfig.proxyStatsMatcher.inclusionRegexps[6]=.*lb_healthy_panic.*' \
  --set 'meshConfig.defaultConfig.proxyStatsMatcher.inclusionRegexps[7]=.*upstream_cx_none_healthy.*'

Not sure about istiod

[andrewgkew]

Thanks @wankai123

Yes I see that in your docs about adding the additional inclusionRegexps to reduce memory footprint. I will add those now

So you are saying I should remove the envoy-stats from the OTEL collector config?

I went with your docs that sends users to the Istio docs on configuring the prometheus receiver

https://istio.io/latest/docs/ops/integrations/prometheus/#option-2-customized-scraping-configurations

Might be worth making it clear on that page to only add istio section not the envoy metrics

Thanks

[wankai123]

yes, you should remove envoy-stats from the OTEL collector config, OAP collects envoy-stats from envoy gRPC metrics service not OTEL. And also need to filter envoy metrics to reduce the OAP memory usage and avoid CPU overhead