Dataflow Templates 2025-04-29-00_RC00

What's Changed

Improvements

• added beam-sdks-java-extensions-sql-expansion-service by @liferoad in #2337

Bug fixes

• Add rowRestriction when fetching BQ schema by @joy91227 in #2273
• Fixing TextImportPipelineIT integration test by @darshan-sj in #2327
• Update docs for inputFilePattern by @manitgupta in #2339
• Add Seperate VPC for Widerow tests by @taherkl in #2335

New Contributors

• @joy91227 made their first contribution in #2273
• @pawankashyapollion made their first contribution in #2300

Full Changelog: 2025-04-15-00_RC01...2025-04-29-00_RC00

Dataflow Templates 2025-04-22-00_RC00

Release week of 2025-04-21

Dataflow Templates 2025-04-15-00_RC01

What's Changed

Improvements

• Moving local spanner io to a different namespace by @darshan-sj in #2231
• Bump timeouts for tests involving FKs/interleaved dependenceis by @Deep1998 in #2239
• Add the support of default_time_zone database option. by @hengfengli in #2255
• Fixing a failing Spanner Integration test by @darshan-sj in #2265
• Move jdbc templates out of googlecloud-to-googlecloud to renamed jdbc-and-googlecloud by @Abacn in #2293
• Spanner Internal failure injector implementation by @darshan-sj in #2277
• Upgrade to Beam 2.64.0 by @Amar3tto in #2297
• Forward migration low priority integration test by @akashthawaitcc in #2309

Bug fixes

• Skip Datastream oracle to spanner IT by @manitgupta in #2250
• Update protobuf version by @damccorm in #2261
• Remove duplicated cdc-agg module by @Abacn in #2276
• Configuring numPartitions for Cassandra and allowing Spanner Mutation BatchSize to be configurable by @VardhanThigle in #2275
• Do not set null Spanner Host in SpannerChangeStreamsToBigQuery Templates by @Abacn in #2286
• Reverse Replication - Addition of missing datatypes to IT by @taherkl in #2303
• fix: Use ignored column families parameter in bigtable to pubsub conn… by @jackdingilian in #2316

New Contributors

• @jackdingilian made their first contribution in #2316

Full Changelog: 2025-03-11-00_RC02...2025-04-15-00_RC01

2025-03-11-00_RC02

What's Changed

Improvements

• Update Dockerfile-template-yaml by @svetakvsundhar in #2222
• Add IF NOT EXISTS clause for spanner ddls used in ITs by @Deep1998 in #2237

Bug fixes

• Fix a bug in CSVToBigQuery where commas in fields are not handled correctly. by @shunping in #2229
• Fixed default DLQ path for Cloud Spanner Change Streams to Big Query by @dedocibula in #2241

Full Changelog: 2025-03-04-00_RC00...2025-03-11-00_RC02

Dataflow Templates 2025-03-04-00_RC00

What's Changed

Improvements

• Support partitioned reads for DateTime column type in JDBC to BigQuery template by @sharan-malyala in #2084
• Add warning about caching with plugin by @derrickaw in #2221
• [DatastreamToSpanner] Spanner Exception handling by @darshan-sj in #2185

Bug fixes

• bug-fix: Use jdbc connection properties for reverse migration by @asthamohta in #2198

New Contributors

• @Rudra-Gujarathi made their first contribution in #2219
• @derrickaw made their first contribution in #2221

Full Changelog: 2025-02-26-00_RC00...2025-03-04-00_RC00

Dataflow Templates 2025-02-26-00_RC00

What's Changed

Improvements

• Change default shadow table prefix of reverse replication template by @manitgupta in #2186
• Remove sdk_container_image example from terraform.tfvars by @manitgupta in #2187
• Upgrade Beam version to 2.63.0 by @jrmccluskey in #2206
• Add load test for cross db txn by @Deep1998 in #2199
• Report Lineage for CsvToBigQuery template by @Abacn in #2205
• Spanner Import/Export INTERLEAVE IN by @jjfox15 in #2128
• Added SCRAM-SHA-512 support to dataflow kafka-to-bigquery by @vgnanasekaran in #2181

Bug fixes

• Fix inf issues in Datastream reader by @dhercher in #2213

New Contributors

• @jjfox15 made their first contribution in #2128
• @vgnanasekaran made their first contribution in #2181

Full Changelog: 2025-02-18-00_RC00...2025-02-26-00_RC00

Dataflow Templates 2025-02-18-00_RC00

What's Changed

Improvements

• Switch to use beam-sdks-java-io-aws2 for KinesisToPubsub by @Abacn in #2182
• Add support for byte arrays and uuid arrays in datastream to sql template. by @claudevdm in #2174
• IT for cross database transaction by @Deep1998 in #2160

Bug fixes

• Support inf and -inf values by @dhercher in #2203
• Fix issues for sequences and identity columns by @hengfengli in #2171
• fix: clean shadow tables by @asthamohta in #2207

Full Changelog: 2025-02-12-00_RC00...2025-02-18-00_RC00

Dataflow Templates 2025-02-12-00_RC00

What's Changed

Improvements

• Added VIRTUAL column support to PG dataflow by @atask-g in #2167
• Add UUID support in import/export templates by @lagarwal1 in #2145

Bug fixes

• Fix handling of empty arrays, json arrays and jsonb arrays. by @claudevdm in #2169
• Remove warning message by @dhercher in #2173

New Contributors

• @lagarwal1 made their first contribution in #2145

Full Changelog: 2025-02-04-00_RC00...2025-02-12-00_RC00

Dataflow Templates 2025-02-04-00_RC00

What's Changed

Improvements

• Search indexes by @n-d-joshi in #2152

Full Changelog: 2025-01-29-00_RC01...2025-02-04-00_RC00

Dataflow Templates 2025-01-29-00_RC01

What's Changed

Improvements

• Switch to Java 17 for all Dataflow Templates by @liferoad in #2050
• Support shadow table in different database by @Deep1998 in #2138
• feat: bump to Neo4j driver v5 by @fbiville in #2136
• Bump Beam to 2.62.0 by @Abacn in #2143
• Added support for Tokenlist columns by @n-d-joshi in #2144
• Support avro arrays for postgres insertion. by @claudevdm in #2154

Bug fixes

• Small Fix for Cassandra bulk e2e test by @VardhanThigle in #2137
• Fix the additional sequence kind options by @hengfengli in #2146
• fix: revamp Neo4j kernel version sniffing by @fbiville in #2148
• fix: account for on-premise Neo4j CalVer version format by @fbiville in #2149

New Contributors

• @claudevdm made their first contribution in #2154

Full Changelog: 2025-01-10-00_RC00...2025-01-29-00_RC01