Filtering targets

[magsad]

Hi, thanks for a great tool, really appreciate it! :) Please forgive me for a silly question, but I'm struggling to filter out some targets from being discovered. I have two ecs clusters and prometheus running for each separately (aka dev/prod). However, the tool is picking up both clusters for both Prometheuses(??) when I add the PROMETHEUS_TARGET="true" as environment variable (as designed of course).

How can I filter the data that is picked up by prometheus ecs discoverer by either cluster name or environment? Is it by setting docker labels?

[trallnag]

Hello @magsad, thanks. And that's actually a pretty good question since there's no way to configure PromED to filter certain clusters. But what you can do is use a resource-level permission in the policy. So instead of allowing all resources "Resource": "*" you can limit it to the ECS cluster you are interested in and a second resource for all EC2 instances. So it would look something like this:

{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "ecs:DescribeClusters",
          "ecs:DescribeContainerInstances",
          "ecs:DescribeTaskDefinition",
          "ecs:DescribeTasks",
          "ecs:ListClusters",
          "ecs:ListContainerInstances",
          "ecs:ListTaskDefinitions",
          "ecs:ListTasks",
          "ec2:DescribeInstances"
        ],
        "Resource": [
          "arn:aws:ecs:<region or *>:<aws_account_id or *>:service/<cluster_name>*",
          "arn:aws:ec2:<region or *>:<aws_account_id or *>:instance/*"
        ]
      }
    ]
  }

I think this is something I'll add to the docs and maybe add a knob to configure it directly from PromED configs (via file or env)

[duganth-va]

Not a silly question! I was debating the same thing. My approach was going to be changing the MARKER = "PROMETHEUS_TARGET" for the two instances of prometheus-ecs-services. I was thinking that I would set MARKER = "PROMETHEUES_TARGET_DEV" and MARKER = "PROMETHEUES_TARGET_PRODUCTION" for the different prometheus-ecs-discover service I have running next to my separate prometheus instance. This was along with IAM permissions. Not sure if this would work well or not. Interested in feedback.

[trallnag]

Hey again, @duganth-va. Yes, this should indeed work well. Though in your case I'd do just both things with the policies alone, I think it's cleaner that way. How to separate between PROD and DEV? As long as all ECS / EC2 resources in you stack have an env tag you can use the condition feature of IAM policies.

Here for stuff on ECS tags: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-using-tags.html
Here condition policies: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_condition.html
Example: https://docs.aws.amazon.com/iot/latest/developerguide/tagging-iot-iam.html

So something like that should work (I haven't tried it):

{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "ecs:DescribeClusters",
          "ecs:DescribeContainerInstances",
          "ecs:DescribeTaskDefinition",
          "ecs:DescribeTasks",
          "ecs:ListClusters",
          "ecs:ListContainerInstances",
          "ecs:ListTaskDefinitions",
          "ecs:ListTasks",
          "ec2:DescribeInstances"
        ],
        "Resource": [
          "arn:aws:ecs:<region or *>:<aws_account_id or *>:service/<cluster_name>*",
          "arn:aws:ec2:<region or *>:<aws_account_id or *>:instance/*"
        ],
        "Condition" : {
          "StringEquals" : {
            "aws:RequestTag/env" : "prod"
          }
        }
      }
    ]
  }

But again, using markers by itself should also work

[duganth-va]

I agree it is cleaner as well, my concern was the api calls would error out but this makes sense. I will try this method as well. Thanks for the feedback!