calico-flow-logs-otlphttp-exporter

Warning

Currently in development, use at your own risk.

Export network flow logs from Calico using OTLP/HTTP. Uses the Direct to Collector approach with the Logs SDK and Logging bridge.

The motivation for this project is to be able to ingest network flow logs from Calico into Log Analysis or SIEM tools using the vendor agnostic OTLP format. The idea for the project originates from this blogpost. More context about the project and why it was made can be found here.

See #Datamodel for payload structure.

Table of contents

• Compatibility
• Installation
  • Helm
• Datamodel
• Demo
• Observability
• Development
  • Prerequisites
  • Setup
  • Running
    • Running as container
    • Running as Helm release
      • Prerequisites
      • Install Helm chart from local directory
• Releasing new versions
  • Releasing new container image versions
  • Releasing new Helm chart versions
• Disclaimer
• Contributors
• License

Compatibility

Calico version	Exporter version
3.30	>= 0.13.3
3.31	>= 0.13.3

Installation

Note

See the official documentation for a list of all environment variables which can be set for the OTLP Log HTTP exporter.

Environment variables:

Environment variable	Description	Required	Default value
GOLDMANE_HOST	Host and port of Goldmane, must be present as SAN in certificate used for mTLS	Yes
CA_CERT_PATH	Path to CA certificate used for mTLS connection to Goldmane	Yes
PRIVATE_KEY_PATH	Path to private key used for mTLS connection to Goldmane	Yes
PUBLIC_CERT_PATH	Path to public certificate used for mTLS connection to Goldmane	Yes
RECONNECT_WAIT_TIME_IN_MILLISECONDS	Amount of milliseconds to wait before attempting to reconnect to Goldmane in the event of connection error	No	5000

Helm

Warning

The default installation uses Goldmanes own certificates for mTLS between calico-flow-logs-otlphttp-exporter and Goldmane, this is not a recommended practice. See the Calico documentation for recommended methods to secure communication using certificates.

See charts/calico-flow-logs-otlphttp-exporter/README.md for a list of all Helm chart values.

1. Add Helm chart repository:

   helm repo add \
     calico-flow-logs-otlphttp-exporter \
     https://fredrickb.github.io/calico-flow-logs-otlphttp-exporter
   helm repo update

2. Install Helm release (see the official documentation for environment variables to set the OTLP/HTTP endpoint in env)

   helm upgrade \
     --install \
     --namespace calico-system \
     # Example setting OTLP endpoint using the OTEL_EXPORTER_OTLP_ENDPOINT environment variable
     --set-string env.OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318" \
     calico-flow-logs-otlphttp-exporter \
     calico-flow-logs-otlphttp-exporter/calico-flow-logs-otlphttp-exporter

Datamodel

The exporter forwards Flow in proto/api.proto payload.

Note

Numerical enums are converted to strings.

Example JSON payload sent to OTLP/HTTP endpoint:

{
  "Key": {
    "sourceName": "longhorn-manager-*",
    "sourceNamespace": "longhorn-system",
    "sourceType": "WorkloadEndpoint",
    "destName": "instance-manager-318f1eac2bc7c775b7a6d2e68e9e2800",
    "destNamespace": "longhorn-system",
    "destType": "WorkloadEndpoint",
    "destPort": "8503",
    "destServiceName": "-",
    "destServiceNamespace": "-",
    "destServicePortName": "-",
    "proto": "tcp",
    "reporter": "Src",
    "action": "Allow",
    "policies": {
      "enforcedPolicies": [
        {
          "kind": "CalicoNetworkPolicy",
          "namespace": "longhorn-system",
          "name": "allow-same-namespace",
          "tier": "default",
          "action": "Allow"
        }
      ],
      "pendingPolicies": [
        {
          "kind": "CalicoNetworkPolicy",
          "namespace": "longhorn-system",
          "name": "allow-same-namespace",
          "tier": "default",
          "action": "Allow"
        }
      ]
    }
  },
  "startTime": "1764970845",
  "endTime": "1764970860",
  "sourceLabels": [
    "app.kubernetes.io/instance=longhorn",
    "app.kubernetes.io/managed-by=Helm",
    "app.kubernetes.io/name=longhorn",
    "app.kubernetes.io/version=v1.8.1",
    "app=longhorn-manager",
    "controller-revision-hash=5bb8c89b95",
    "helm.sh/chart=longhorn-1.8.1",
    "longhorn.io/admission-webhook=long horn-admission-webhook",
    "longhorn.io/conversion-webhook=longhorn-conversion-webhook",
    "longhorn.io/recovery-backend=longhorn-recovery-backend",
    "pod-template-generation=2"
  ],
  "destLabels": [
    "longhorn.io/component=instance-manager",
    "longhorn.io/data-engine=v1",
    "longhorn.io/instance-manager-image=imi-7d4dc4d4",
    "longhorn.io/instance-manager-type=aio",
    "longhorn.io/managed-by=longhorn-manager",
    "longhorn.io/node=k8s-worker-2"
  ],
  "packetsIn": "11",
  "packetsOut": "14",
  "bytesIn": "957",
  "bytesOut": "1226",
  "numConnectionsStarted": "1",
  "numConnectionsCompleted": "1",
  "numConnectionsLive": "1"
}

Demo

Search logs in Grafana using Loki as a Datasource after ingesting logs to the Loki OTLP endpoint from OpenTelemetry Collector.

Note

Using OpenTelemetry Collector as layer between calico-flow-logs-otlphttp-exporter and Loki is optional, you can point the exporter directly at Lokis OTLP endpoint.

Observability

There is a Loki Grafana dashboard which can be imported to view the state of flows when logs are sent to a Loki OTLP endpoint.

Development

Note

See the official documentation for a list of all environment variables which can be set for the OTLP Log HTTP exporter.

Prerequisites

• go >= 1.25
• protoc, 3.21.12
• make
• kubectl
• base64
• docker, 28.2.1
• helm >= 3.x
• Calico (installation documentation)
• Goldmane running in namespace calico-system (installation documentation)
• dlv (optional, for debugging, installation documentation)
• k3d (optional, for setting up a local development cluster, installation documentation)

Setup

1. Install development tools: make install-development-packages
2. Fetch the protobufs from the Calico project: make fetch-protobuf-definition
3. Generate code from protobufs make generate-code-from-protobuf
4. Add line to /etc/hosts in order to be able to use Goldmane certs for running locally:

   127.0.0.1 goldmane
   

5. (Optional, requires docker and k3d) setup k3d cluster make setup-k3d [K3D_CLUSTER_NAME=<cluster-name>] [K3D_CLUSTER_CALICO_VERSION=<calico-version>]
6. (Optional, requires k3d cluster to be created) install calico to k3d cluster make install-calico [K3D_CLUSTER_CALICO_VERSION=<calico-version>]

Running

Warning

For development we copy the certificates from a Goldmane deployment running in the cluster directly. Do not use this approach in production environments, this is just for development.

1. (Optional) start k3d cluster make start-k3d [K3D_CLUSTER_NAME=<cluster-name>] [K3D_CLUSTER_CALICO_VERSION=<calico-version>]
2. Port-forward the Goldmane service: make port-forward-goldmane [GOLDMANE_NAMESPACE=<goldmane namespace>]
3. Copy the certificates from a running instance of Goldmane: make copy-goldmane-certs-from-kubernetes-deployment [GOLDMANE_NAMESPACE=<goldmane namespace>]
4. Start the otel-collector, Loki and Grafana: make docker-compose-up
5. Run project: make run [OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:3100/otlp] (by default OpenTelemetry Collector is used, you can override to use Loki directly)
   1. Optionally, run make debug [OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:3100/otlp] to debug using dlv
6. Open Grafana Explore with Loki search + JSON parsing enabled
   • Username: admin123
   • Password: admin123
7. (Optional) stop k3d cluster make stop-k3d [K3D_CLUSTER_NAME=<cluster-name>] [K3D_CLUSTER_CALICO_VERSION=<calico-version>]

Running as container

1. (Optional) start k3d cluster make start-k3d [K3D_CLUSTER_NAME=<cluster-name>] [K3D_CLUSTER_CALICO_VERSION=<calico-version>]
2. Port-forward the Goldmane service: make port-forward-goldmane [GOLDMANE_NAMESPACE=<goldmane namespace>]
3. Copy the certificates from a running instance of Goldmane: make copy-goldmane-certs-from-kubernetes-deployment [GOLDMANE_NAMESPACE=<goldmane namespace>]
4. Start the otel-collector, Loki and Grafana: make docker-compose-up
5. Build container image: make build-container-image [TAG=<tag>]
6. Run the container image: make run-container
7. (Optional) stop k3d cluster make stop-k3d [K3D_CLUSTER_NAME=<cluster-name>] [K3D_CLUSTER_CALICO_VERSION=<calico-version>]

Running as Helm release

Prerequisites

1. OpenTelemetry-Collector installed:
   1. Namespace must be opentelemetry-collector
   2. Name of Service for OpenTelemetry-Collector must be opentelemetry-collector
2. (Optional) Adapt values in hack/charts/calico-flow-logs-otlphttp-exporter/override.yaml

Install Helm chart from local directory

1. Install Helm release:

   make install-helm-chart-from-local-dir

Releasing new versions

When releasing new versions, it has to be done in 2 steps:

1. Release new container image version
2. Release new Helm chart version

Releasing new container image versions

Always update CHANGELOG.md when releasing a new container image version

Upon merge to main the workflow .github/workflows/release_container_image.yaml builds a new container image. Version is based on the collection of conventional commits between previous release and current commit.

All container image versions are here.

Proceed to Releasing new Helm chart versions afterwards.

Releasing new Helm chart versions

Always update charts/calico-flow-logs-otlphttp-exporter/CHANGELOG.md when releasing a new Helm chart version

Bumping of Helm chart versions is done manually in the charts/calico-flow-logs-otlphttp-exporter/Chart.yaml. Upon merge to main the workflow .github/workflows/release_charts.yaml packages and uploads the new helm chart version to branch gh-pages.

Disclaimer

This project is a personal open-source initiative and is not affiliated with, endorsed by, or associated with any of my current or former employers. All opinions, code, and documentation are solely those of myself and the individual contributors.

The project is not affiliated with Project Calico or any of its subsidiaries. The use of the Calico name and/or logo is for informational purposes only and does not imply any endorsement or affiliation with the Calico project.

Contributors

Erik	Fredrick Biering

License

calico-flow-logs-otlphttp-exporter is distributed under the MIT License.