Celery Integration for Poolboy

[marcosmamorim]

Overview

This PR introduces Celery-based distributed task processing for Poolboy, enabling horizontal scalability and improved performance through asynchronous background processing of resource management operations.

Motivation

Poolboy supports a manager mode architecture where multiple operator replicas distribute event handling across pods. This approach provides good reliability and basic load distribution.

Celery integration complements this architecture by adding:

• True parallel processing - Heavy operations (templating, provisioning) run in dedicated worker processes without blocking event handling
• Independent scaling - Workers scale separately from the operator via HPA, responding dynamically to workload
• Task prioritization - Different queue configurations for different operation types
• Enhanced observability - Per-task metrics, execution tracking, and monitoring via Flower dashboard
• Robust retry handling - Automatic retry with exponential backoff for transient failures

---

Solution Architecture

The implementation adds three new Kubernetes components alongside the existing Poolboy operator:

1. Redis - Message broker and result backend, also used for distributed locks and shared cache
2. Celery Workers - Execute resource management tasks in parallel
3. Celery Beat Scheduler (optional) - Periodic task scheduling for reconciliation

Processing Flow

When workers are enabled:

1. The operator receives Kubernetes events via Kopf handlers
2. Instead of processing directly, it dispatches a Celery task with the resource definition
3. Workers pick up tasks from partitioned queues and execute the same business logic
4. Distributed locks ensure only one worker processes a given resource at a time
5. Periodic maintenance uses Celery's group primitive to batch process multiple resources in parallel

The operator falls back to synchronous processing when workers are disabled, ensuring backward compatibility.

---

Key Features

Feature Flags per Resource Type

Each resource class can be individually enabled for worker processing:

• useWorkers.resourcePool.enabled
• useWorkers.resourceHandle.enabled
• useWorkers.resourceClaim.enabled

This allows gradual migration and rollback per component.

Operation Modes

Three modes control how periodic reconciliation is handled:

Mode	Description	Use Case
scheduler	Celery Beat dispatches periodic tasks	Default - recommended
daemon	Kopf daemons dispatch to workers	Lower latency
both	Both mechanisms run	High availability

Partitioned Queues

Tasks are routed to partitioned queues using consistent hashing of namespace/name:

• Same resource → same partition → sequential processing (event ordering preserved)
• Different resources → parallel processing across partitions and workers
• Configurable partition count per resource type

Distributed Locking

Redis-based distributed locks prevent concurrent processing of the same resource across workers:

• Lock acquired before execution, released after
• Tasks retry with exponential backoff if lock is unavailable
• Lock timeout prevents deadlocks from crashed workers

Shared Redis Cache

A unified cache system enables state sharing between operator and workers:

• Object caches (ResourceHandle, ResourceClaim, ResourcePool) stored in Redis
• Workers can access cached objects without API round-trips
• Eliminates the need for separate "processed" markers
• Automatic backend selection: in-memory for standalone, Redis for distributed

Horizontal Pod Autoscaler

Workers support HPA based on CPU/memory utilization for automatic scaling under load.

---

Components Migrated

Component	Status	Notes
ResourcePool	Complete	Pool maintenance via workers
ResourceHandle	Complete	Handle provisioning and lifecycle management via workers
ResourceClaim	Complete	Claim reconciliation via workers (with binding caveat)
ResourceProvider	—	Static configuration, no worker benefit
ResourceWatch	—	Watch management remains in operator

Note: ResourceClaim binding still occurs in the operator to leverage in-memory cache for pool handle discovery. Once bound, subsequent updates are processed by workers.

---

Configuration

All configuration is managed through Helm values.yaml:

# Enable worker infrastructure
worker:
  enabled: true
  replicas: 2
  config:
    concurrency: 4
    pool: prefork

# Enable workers per resource type
useWorkers:
  resourcePool:
    enabled: true
    daemonMode: "scheduler"
    partitions: 2
  resourceHandle:
    enabled: true
    daemonMode: "scheduler"
    partitions: 4
  resourceClaim:
    enabled: true
    daemonMode: "scheduler"
    partitions: 4

# Redis for message broker and cache
redis:
  enabled: true
  persistence:
    enabled: true

# Optional: Celery Beat scheduler for periodic tasks
scheduler:
  enabled: true

---

Observability

• Task execution tracked via Celery's built-in mechanisms
• Flower dashboard available for task monitoring (optional deployment)
• Redis metrics available via standard Redis exporters
• Worker pod metrics for HPA scaling decisions