Skip to content

stage_external_sources creates source delta tables if they don't exist. #353

@rameshkanjinghat

Description

@rameshkanjinghat

I’m using dbt with Spark sessions to run my transformations. My source tables are Delta tables stored in ADLS Gen2.

When I run stage_external_sources, it creates empty delta tables in the source location, if they don’t already exist. This results in a folder named after the source, containing only the _delta_log directory with json file.

Is this expected behavior?

Below is the content of the JSON file inside the _delta_log folder:

{"commitInfo":{"timestamp":1744449272223,"operation":"CREATE TABLE","operationParameters":{"isManaged":"false","description":null,"partitionBy":"[]","properties":"{}"},"isolationLevel":"Serializable","isBlindAppend":true,"operationMetrics":{},"engineInfo":"Apache-Spark/3.4.0 Delta-Lake/2.4.0","txnId":"85234e65-6d2c-42f9-a817-5bd5ba4bfe42"}}
{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}
{"metaData":{"id":"72fcb134-0b8b-431d-a725-35bed52fe040","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[]}","partitionColumns":[],"configuration":{},"createdTime":1744449272165}}

Thanks!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions