Skip to content

feat: add AWS Data Pipeline support#1036

Open
james00012 wants to merge 1 commit intomasterfrom
feat/add-data-pipeline-support
Open

feat: add AWS Data Pipeline support#1036
james00012 wants to merge 1 commit intomasterfrom
feat/add-data-pipeline-support

Conversation

@james00012
Copy link
Contributor

@james00012 james00012 commented Mar 7, 2026

Summary

  • Add data-pipeline resource type to discover and delete AWS Data Pipeline resources
  • Extends util.ParseTimestamp with bare ISO 8601 format fallback (2015-01-01T00:00:00) used by the Data Pipeline API
  • DescribePipelines calls batched in groups of 25 per AWS API limit
  • Config key DataPipeline supports names_regex and time filtering

Files changed

File Change
aws/resources/data_pipeline.go New resource implementation
aws/resources/data_pipeline_test.go Unit tests (7 subtests + delete)
aws/resource_registry.go Register in regional resources
config/config.go Add DataPipeline field + allResourceTypes()
config/config_test.go Add to emptyConfig() helper
util/time.go Add bare ISO 8601 fallback to ParseTimestamp
util/time_test.go Test bare ISO 8601 parsing
docs/supported-resources.md Document new resource
go.mod / go.sum Add datapipeline SDK dependency

Test plan

  • go test ./aws/resources -run "TestListDataPipelines|TestDeleteDataPipeline" -v — all 8 tests pass
  • go test ./util/... -run TestParseTimestamp -v — all 3 format tests pass
  • go test ./config/... -run "TestAllResourceTypesComplete|TestConfig_Empty" -v — 129 fields match
  • cloud-nuke inspect --resource-type data-pipeline against account 087285199408
  • cloud-nuke nuke --resource-type data-pipeline --region us-west-2 to clean up sample pipeline

@james00012 james00012 requested a review from denis256 as a code owner March 7, 2026 16:33
@james00012 james00012 force-pushed the feat/add-data-pipeline-support branch 2 times, most recently from f38af4e to eb44d17 Compare March 7, 2026 16:40
denis256
denis256 previously approved these changes Mar 8, 2026
gcagle3
gcagle3 previously approved these changes Mar 9, 2026
Add ability to discover and delete AWS Data Pipeline resources.
This enables cleanup of stale pipelines (e.g., sample-pipeline from 2015)
during scheduled cloud-nuke runs.

- New resource type `data-pipeline` with list, describe, and delete support
- DescribePipelines batched in groups of 25 (API limit)
- Extend util.ParseTimestamp with bare ISO 8601 fallback for Data Pipeline timestamps
- Config key: `DataPipeline`, supports name regex and time-based filtering
@james00012 james00012 dismissed stale reviews from gcagle3 and denis256 via 0f67d72 March 9, 2026 19:19
@james00012 james00012 force-pushed the feat/add-data-pipeline-support branch from eb44d17 to 0f67d72 Compare March 9, 2026 19:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants