Metric for exceeding limits

[dtwilliamsWork]

Is there a specific Karpenter metric I can use to monitor if a node pool's limit is exceeded?
https://karpenter.sh/docs/reference/metrics/

I would like to set up a Prometheus alert manager rule to monitor it.

If not, what is the error message that displays in the Karpenter logs to look out for?
I can set up a CloudWatch metric instead if its not possible with Karpenter metrics.

[dtwilliamsWork]

I've managed to test this myself and found the relevant message in the logs.

all available instance types exceed limits for nodepool:

Our Karpenter pod logs are exported to CloudWatch using FluentBit so we're able to add a metric filter on our log group with a relevant alarm

Here's the Terraform code I used, if it's helpful for anyone. I've configured a SNS topic to send our alerts to, which creates them on our OpsGenie platform.

resource "aws_cloudwatch_log_metric_filter" "karpenter_node_limit_exceeded" {
  name           = "Karpenter Node Limits Exceeded - ${var.cluster_name}"
  log_group_name = "/aws/containerinsights/${var.cluster_name}/application"
  pattern        = "{ $.log_processed.error = %all available instance types exceed limits for nodepool:% && $.kubernetes.pod_name = %karpenter% }"

  metric_transformation {
    name          = "Karpenter Node Limits Exceeded - ${var.cluster_name}"
    namespace     = "Karpenter"
    value         = "1"
    default_value = "0"
    unit          = "Count"
  }

  depends_on = [
    kubectl_manifest.fluent-bit-daemonset
  ]
}

data "aws_sns_topic" "opsgenie-alerts" {
  name = "opsgenie-alerts"
}

resource "aws_cloudwatch_metric_alarm" "karpenter_node_limit_exceeded" {
  alarm_name        = "karpenter_node_limit_exceeded_alarm_${var.cluster_name}"
  alarm_description = "Alarm triggered by Cloudwatch Logs for Karpenter Node Limits Exceeded"
  alarm_actions     = [data.aws_sns_topic.opsgenie-alerts.arn]

  metric_name         = "Karpenter Node Limits Exceeded - ${var.cluster_name}"
  threshold           = 0
  statistic           = "Sum"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 5
  period              = 60
  namespace           = "Karpenter"

  depends_on = [
    kubectl_manifest.fluent-bit-daemonset
  ]
}