Use Mesos task labels as prometheus labels

[StephanErb]

Since version 0.22 Mesos is support task labels. (https://mesosphere.com/blog/2015/03/27/mesos-0-22-0-released/)

...
{
  "executor_id": "default",
  "framework_id": "20150312-120017-16777343-5050-39028-0000",
  "id": "3",
  "labels": [
    {
      "key": "environment",
      "value": "prod"
    },
    {
      "key": "bananas",
      "value": "apples"
    }
  ],
  "name": "Task 3",
  "slave_id": "20150312-115625-16777343-5050-38751-S0",
  "state": "TASK_FINISHED",
  ...
},

Labels are also available in /monitor/statistics.json. We should attach those labels to our the scraped tasks metrics.

[brian-brazil]

We need to be a little careful here, what a framework has as labels may not be the labels it has in a broader context. I'd be tempted to wait a bit and see how this feature is used first. If there's any way the end user of a framework (as distinct to the owner of the mesos system) can control these labels, then it's not safe to blindly apply them to everything as it may mess up the monitoring of the mesos system.

[StephanErb]

My intention was exactly that. See the Aurora review request: https://reviews.apache.org/r/35990/. As far as I know Marathon is already supporting user-defined task labels.

Isn't it somewhat intentional that an enduser can label his tasks to easily group and monitor them?

[brian-brazil]

There's a difference between monitoring a user's tasks, and monitoring the mesos cluster. The mesos_exporter is about monitoring the mesos cluster. So while exposing these labels in some way is a good idea, applying these labels to all metrics that are exposed for a task doesn't make sense in the general case.

Labels provided by users may not be what whoever runs the mesos cluster wants, as the cluster admin may care about different dimensions and want to avoid rogue labels.

[StephanErb]

There's a difference between monitoring a user's tasks, and monitoring the mesos cluster. The mesos_exporter is about monitoring the mesos cluster.

I am not total convinced here, given that the mesos_exporter exposes information about task resource usage and limits. That is definitely something a user needs to be aware about (e.g., when sizing his tasks).

[brian-brazil]

The user needs to be aware of that, and seeing that is a service provided by the cluster admin.

The general point here is that you need to be very careful about what labels you apply to a target, and allowing end-users to control that rather than the person running the monitoring system is asking for trouble.

[StephanErb]

Would a configurable label whitelist mitigate your concerns?

[brian-brazil]

That might work. We need something that works for all reasonable deployment strategies. A setup where the user and the cluster admin are one and the same (which I'm guessing you have, and I'd have something similar) and thus that the user can be trusted not to do something silly is only one possible deployment strategy.