Observability Revamp Roadmap

[srilman]

For Daft to become the data processing engine for AI models and multimodal data, it must be:

• Easy to use, with a quick responsive iteration cycle for developers
• Robust and fault-tolerant, especially at the scale of 100's of nodes or running for days at a time
• Performant

As one aspect of these goals, it should be effortless for a user to see what their query is "doing". Our idea is to provide a dashboard-like experience, aka the Daft Dashboard, that can have:

• Centralized tracking across multiple queries
• UI for monitoring, visualization, tracking query progress & metrics, looking at logs or stdout, error notifications, outputs, etc
• An easy deployment strategy, available with pip install daft
• Compatibility with different environments (Kubernetes, serverless, notebooks)
• Minimal overhead

It can be similar to the Dask Dashboard, but with a focus on mapping 1-1 to user code and less of internals. In some situations, the Daft Dashboard could supersede or power existing observability tooling, like the progress bars, the EXPLAIN ANALYZE Markdown, .explain for plans, and so on.

⌛️🗡️🐟 Milestone 1: v0 Swordfish Observability

Status: ✅ Completed

Goal: Provide a standalone dashboard setup for single-node deployments

Timeline: v0.6.2 v0.6.6

• Only Native runner, useful for serverless-like setups
• Expose basic status, metrics, logging, error tracking, display for DataFrame output, etc
• Integrate in Jupyter environment and replace existing progress bar

Tasks: Engine

• Construct the Query Subscriber Framework for Swordfish Queries
• Extend the framework to capture unoptimized plan, optimized plan, and results
• Extend the framework to capture stats as a RuntimeSubscriber

Tasks: Dashboard Server

• daft dashboard CLI with Daft Python package
• Refactor server architecture to use Axum
• Refactor query metadata endpoint
• Refactor DataFrame output endpoint
• Metrics support

Tasks: UI

• Refactor Queries Page with up-to-date status indicators
• Refactor query page with high level metadata display
• Logical plan ASCII display in UI
• Embed progress bar / metrics in query page
• Basic DataFrame output in query page

🐟⚔️🐟 Milestone 2: v0 Flotilla Observability

Status: ⌛️ In Progress

Goal: Basic query and metrics observability for distributed engine deployments

Timeline: v0.7.1

Tasks: Flotilla

• Track runtime metrics tracking for Flotilla pipeline nodes
• Emit Flotilla metrics through OpenTelemetry (OTEL)
• Basic metric aggregation for all operator nodes (rows in, rows out, CPU runtime in microseconds)

Custom metrics aggregation for special nodes

• Sinks (files written, bytes written)
• UDF (custom metrics)
• Filter (selectivity)
• Explode (expansion?)
• Limit (handle the 2 stage limit)
• Repartition (handle the 2 stage repartition)
• Hash & Broadcast Join (handle the 2 stage join)
• Sort (handle the pre-sort, sampling, redistribution, and post-sort)
• Into Partitions / Into Batches (special logic)
• Pivot (handle the initial groupby, then later pivot)
• Cross Join (handle the broadcasting)

Tasks: Dashboard-Related

• Support the subscriber framework within Flotilla
• Track runner type in dashboard

🗡️🐟 Milestone 3: v1 Swordfish Observability

Status: 💤 Not Started

Goal: Extend the Swordfish dashboard integration to cover more cases

Timeline: TODO

• Emit Swordfish metrics through OTEL
• Internal server request / response logging
• Add logging tracking in dashboard server
• Display logging panel in UI
• Enable Dashboard startup and embedding in Jupyter notebooks
• Replace TQDM Jupyter progress bar with dashboard iframe
• Replace DataFrame viz in Jupyter with dashboard iframe
• ASCII-formatted physical plan in query page
• Properly handle subscriber cleanup for failures and cancels
• Real-time status monitoring, especially for errors / failures and cancels

🐟⚔️🐟 Milestone 4: v1 Flotilla Observability

Status: 💤 Not Started

Goal: Extend support for Flotilla observability

Timeline: TODO

Tasks: Flotilla

• Logging transmission
• Any custom metrics for special operators (as listed above) from Flotilla info

Investigation

Ideas that we are currently looking into. Once ready, we will add some of these ideas to the roadmap. Please let us know if you believe any of these (or anything else) is important to prioritize.

• Hardware Utilization. Seems possible in some distributed setups but difficult with Swordfish, especially without having full control of the node
• Query visualization in DAGs
• Flotilla task-level monitoring, especially retries
• Flotilla engine-level statistics, shuffle utilization, etc
• Single progress percentage. Shouldn't be too difficult for map-only pipelines, but unusual for non-map only
• Persistence: Explore using https://fjall-rs.github.io for KV embedded store

[srilman]

I updated parts of the roadmap to include additional requirements around connecting to external metrics systems, such as OpenTelemetry and Prometheus. In addition, I've simplified portions of the distributed roadmap based on initial implementation

[VOID001]

Added an issue for the current dashboard #5518