Skip to content

Dagster Housekeeping #3956

Open
Open
@zaneselvans

Description

@zaneselvans

Description

It's been ~1 year since we've worked intensively on orchestration stuff, and Dagster has evolved since then. There are deprecation and design changes that we probably need to accommodate as the project continues to grow in complexity. Is there anything we want to bring in from #2386? Or can that be closed?

- [ ] #3161
- [ ] https://github.com/catalyst-cooperative/pudl/issues/3977
- [ ] Remove "experimental" warning from Parquet IO Manager
- [ ] Look at using Dagster's [tags](https://docs.dagster.io/concepts/metadata-tags/tags#tags) and hierarchical [key prefixes](https://docs.dagster.io/concepts/assets/software-defined-assets#multi-component-asset-keys) to better organize our DAG
- [ ] Switch to using Postgres instead of SQLite for local development.
- [ ] #3928
- [ ] #3913
- [ ] Investigate whether we can use Dagster's built-in partitioning to split up big jobs like EPA CEMS more simply.
- [ ] https://github.com/catalyst-cooperative/pudl/issues/2468
- [ ] https://github.com/catalyst-cooperative/pudl/issues/2470

Metadata

Metadata

Assignees

No one assigned

    Labels

    dagsterIssues related to our use of the Dagster orchestratorepicAny issue whose primary purpose is to organize other issues into a group.

    Type

    No type

    Projects

    Status

    Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions