-
Notifications
You must be signed in to change notification settings - Fork 132
Open
Description
I’m using dbt with Spark sessions to run my transformations. My source tables are Delta tables stored in ADLS Gen2.
When I run stage_external_sources, it creates empty delta tables in the source location, if they don’t already exist. This results in a folder named after the source, containing only the _delta_log directory with json file.
Is this expected behavior?
Below is the content of the JSON file inside the _delta_log folder:
{"commitInfo":{"timestamp":1744449272223,"operation":"CREATE TABLE","operationParameters":{"isManaged":"false","description":null,"partitionBy":"[]","properties":"{}"},"isolationLevel":"Serializable","isBlindAppend":true,"operationMetrics":{},"engineInfo":"Apache-Spark/3.4.0 Delta-Lake/2.4.0","txnId":"85234e65-6d2c-42f9-a817-5bd5ba4bfe42"}}
{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}
{"metaData":{"id":"72fcb134-0b8b-431d-a725-35bed52fe040","format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[]}","partitionColumns":[],"configuration":{},"createdTime":1744449272165}}
Thanks!
Metadata
Metadata
Assignees
Labels
No labels