Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

documentation: quantities and metric relation, case sensitiveness, metric context/selection #101

Open
cforce opened this issue Jun 16, 2021 · 2 comments

Comments

@cforce
Copy link

cforce commented Jun 16, 2021

quantities and metric relation

Taken from the docs..
"They are specified as Quantities, so you can use m | "" | k | M | G | T | P | E to easily represent the value you want to use."

The above sentence quotes that highWatermark and lowWatermark values can or must? be extended by quantity.
I assume its dependent on the type of metric.

What type of quantity is "" ?
What about percent values like i think it shall be able to use all kind of metrics which have a defined max and a min value like e.g kubernetes.memory.usage_pct or kubernetes.cpu.usage_pct. Do i need to set only the value 90 or 90% ?

If i don't add a quantity but just the number, will it take a default (which) or will it fail?

case sensitiveness

There are different metrics i can use . eg.

The docker Metrics e.g "Memory" have upper case. Is there generally any case sensitiveness ?

metric context/selection

Taking above two metric sources - system/docker and kubernetes - i could use
similar metrics from both e.g system.mem.used vs kubernetes.memory.usage
If i deploy a resource wpa

      metricSelector:
        matchLabels:
          app: {{ .Chart.Name }}
          release: {{ .Release.Name }}

will the both metric measurements be scoped to the container level per pod or do they have different scopes and which?

Which of these might be the better choice to control the scale

@CharlyF
Copy link
Contributor

CharlyF commented Jun 17, 2021

Hi there!

The above sentence quotes that highWatermark and lowWatermark values can or must? be extended by quantity.

They can. They will be correctly interpreted whether you specify a Quantity identifier (m, G, k ...) as can be seen here.
So concretely using the following is equivalent:

  metrics:
  - external:
      highWatermark: "70"
      lowWatermark: "60"
      metricName: sinus
      metricSelector:
        matchLabels:
          foo: bar

to

  metrics:
  - external:
      highWatermark: "70000m"
      lowWatermark: "60"
      metricName: sinus
      metricSelector:
        matchLabels:
          foo: bar

What type of quantity is "" ?

If you don't specify anything the value will be just be an int64. The conversion in the code makes it so if you don't specify a quantity and the value will be converted to a quantity and be compared as a millivalue (so 70 becomes 70000m) and compared to the value returned by the metrics server.


For context, the smallest value that can be represented is 1 milli, anything below will be 0.
The External Metrics server exposes a Quantity type, so if you query it directly you will see the value converted accordingly. Example:

➜  ~ kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/datadogmetric@datadog-agent:clustercheck-worker" | jq .
{
  "kind": "ExternalMetricValueList",
  "apiVersion": "external.metrics.k8s.io/v1beta1",
  "metadata": {},
  "items": [
    {
      "metricName": "datadogmetric@datadog-agent:clustercheck-worker",
      "metricLabels": null,
      "timestamp": "2021-06-17T15:26:30Z",
      "value": "39658165378n"
    }
  ]
}

Then the autoscaler client used in the Watermark Pod Autoscaler controller is from kubernetes upstream and converts the value into an int64 as a milli, per here. As you can see, all methods in the interface return an int64.
So the above example will be converted to 39658165 and used in the WPA controller as a milli later in the code.

What about percent values like i think it shall be able to use all kind of metrics which have a defined max and a min value like e.g kubernetes.memory.usage_pct or kubernetes.cpu.usage_pct. Do i need to set only the value 90 or 90% ?

I think we retrieve a float (so 0.9) from the backend, which aligns with "Shown as fraction". The cluster agent will convert this to a Quantity (from first receiving/storing the float from the API and then converting it to a quantity), so if queried the cluster agent will return 900m. Which will then become 900 from the upstream podautoscaler client and be converted back to 900m in the WPA controller (as detailed above).

I would therefore recommend using 950m as high watermark to represent 95% for instance (or whatever your watermark might be).

The docker Metrics e.g "Memory" have upper case. Is there generally any case sensitiveness ?

I don't want to misinform you on this one, I don't see upper cases in the officially collected docker metrics, but I might be missing something. I would expect everything to be lowercased but could be wrong (if the above link does not clarify the metrics, I can circle back and confirm internally).

will the both metric measurements be scoped to the container level per pod or do they have different scopes and which?

If you don't scope over a specific container with the docker metric or a specific pod with the kubernetes one, you will end up with an average of what the metric represents.
The best practice is to visualise the metric in the metrics explorer and see how the value changes depending on the scope so I make sure I set my watermarks correctly.

Which of these might be the better choice to control the scale

Hard to say without having the full context of your application and infrastructure, what I would say is that docker metrics are direct interfaces of the cgroups whereas the kubernetes ones are interfaced by the CRI which itself interfaces the cgroups.

I hope this clarifies your questions!

@CharlyF
Copy link
Contributor

CharlyF commented Jun 28, 2021

Hi there - Circling back on this issue. Did my response help?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants