Description
Problem Statement
- I typically run my services in an environment, and begin collecting traces, before I have defined any boards or triggers in my Terraform code.
- Honeycomb creates new datasets and columns on the fly
- If I deploy to some environments but not others, the Terraform code that I write to create boards and triggers may become valid to run in some environments, but not others, because it refers to columns or datasets that do not exist
- If, instead, I try to proactively create columns and datasets using Terraform, then that code be invalid in any environment where I have deployed
I'm forced to literally ship every change to production first, and then create a separate pull request later on that adds observability. This is a waste of my time and creates an observability gap.
Proposed Solution
For boards and columns, the state is trivial and I would argue that this provider should break ranks with Terraform and provide idempotent/upsert semantics. In other words: if the board or column already exists and if it is identical to the resource specification, the provider should simply pretend that create succeeded vs. erroring because the resource already exists.
Workarounds
One can switch to "observe first" i.e. decide on column and dataset names, ship a Terraform change to all environments, and then work on the service in question. That's alienating however.