v0.67.8
Lots of improvements the past few weeks! We've continued to focus on improving performance and reliability, and have made some significant progress across the board.
What's Changed
Streaming in V1
We've added first-class streaming support in V1. It works the same as it did in V0 (see the put_stream methods on the Context for pushing stream events), and the stream method on the RunRefs for consuming them. We've also re-published the streaming docs.
Tenant Prometheus Metrics
Our last release highlighted global Prometheus metrics, and we now support scraping metrics for individual tenants. This means you can monitor the performance of a specific tenant in a multi-tenant environment.
Available metrics:
| Metric Name | Type | Description |
|---|---|---|
hatchet_tenant_workflow_duration_milliseconds |
Histogram | Duration of workflow execution in milliseconds (DAGs and single tasks) |
hatchet_tenant_queue_invocations_total |
Counter | The total number of invocations of the queuer function |
hatchet_tenant_created_tasks_total |
Counter | The total number of tasks created |
hatchet_tenant_retried_tasks_total |
Counter | The total number of tasks retried |
hatchet_tenant_succeeded_tasks_total |
Counter | The total number of tasks that succeeded |
hatchet_tenant_failed_tasks_total |
Counter | The total number of tasks that failed (in a final state, not including retries) |
hatchet_tenant_skipped_tasks_total |
Counter | The total number of tasks that were skipped |
hatchet_tenant_cancelled_tasks_total |
Counter | The total number of tasks cancelled |
hatchet_tenant_assigned_tasks |
Counter | The total number of tasks assigned to a worker |
hatchet_tenant_scheduling_timed_out |
Counter | The total number of tasks that timed out while waiting to be scheduled |
hatchet_tenant_rate_limited |
Counter | The total number of tasks that were rate limited |
hatchet_tenant_queued_to_assigned |
Counter | The total number of unique tasks that were queued and later got assigned to a worker |
hatchet_tenant_queued_to_assigned_time_seconds |
Histogram | Buckets of time in seconds spent in the queue before being assigned to a worker |
hatchet_tenant_reassigned_tasks |
Counter | The total number of tasks that were reassigned to a worker |
Events Frontend
We've brought back the frontend for the events page, where you can see recent events and the runs they've triggered.
Other New Features
- Added support for setting a
CRON_TZ=<<timezone>>prefix when creating crons so they run in a specific timezone. For instance,CRON_TZ=America/Chicago 0 0 * * *will run the cron at midnight in Chicago time.
Bug Fixes and Performance Improvements
- Fixed an issue where replaying a large number of workflows in bulk would fail.
- Fixed a number of issues with namespaces on the SDKs.
- Fixed issues with invalid CEL expressions causing event processing issues.
- Fixed a few resource leakage problems causing performance issues, especially in hatchet-lite