Skip to content

Conversation

@ebyhr
Copy link
Member

@ebyhr ebyhr commented Nov 28, 2024

Description

This table will help us debug issues in transaction logs.
I didn't document it because it's mainly for debug purpose and I expect most users don't use it in normal situations.

trino> SELECT * FROM delta.tpch."region$transactions";

 version |                                                                                                                                                                                      transaction                                       >
---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------->
       0 | [ {                                                                                                                                                                                                                                    >
         |   "commitInfo" : {                                                                                                                                                                                                                     >
         |     "version" : 0,                                                                                                                                                                                                                     >
         |     "timestamp" : 1733023200751,                                                                                                                                                                                                       >
         |     "userId" : "user",                                                                                                                                                                                                                 >
         |     "userName" : "user",                                                                                                                                                                                                               >
         |     "operation" : "CREATE TABLE AS SELECT",                                                                                                                                                                                            >
         |     "operationParameters" : {                                                                                                                                                                                                          >
         |       "queryId" : "20241201_032000_00022_p82tv"                                                                                                                                                                                        >
         |     },                                                                                                                                                                                                                                 >
         |     "clusterId" : "trino-testversion-db504efc-1f50-4f2d-9605-a57eda620b9d",                                                                                                                                                            >
         |     "readVersion" : 0,                                                                                                                                                                                                                 >
         |     "isolationLevel" : "WriteSerializable",                                                                                                                                                                                            >
         |     "isBlindAppend" : true                                                                                                                                                                                                             >
         |   }                                                                                                                                                                                                                                    >
         | }, {                                                                                                                                                                                                                                   >
         |   "protocol" : {                                                                                                                                                                                                                       >
         |     "minReaderVersion" : 1,                                                                                                                                                                                                            >
         |     "minWriterVersion" : 2                                                                                                                                                                                                             >
         |   }                                                                                                                                                                                                                                    >
         | }, {                                                                                                                                                                                                                                   >
         |   "metaData" : {                                                                                                                                                                                                                       >
         |     "id" : "ccb5b6f1-4efd-46b8-a212-4c3d5dc8fec2",                                                                                                                                                                                     >
         |     "format" : {                                                                                                                                                                                                                       >
         |       "provider" : "parquet",                                                                                                                                                                                                          >
         |       "options" : { }                                                                                                                                                                                                                  >
         |     },                                                                                                                                                                                                                                 >
         |     "schemaString" : "{\"type\":\"struct\",\"fields\":[{\"name\":\"regionkey\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"name\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"comment\",\"t>
         |     "partitionColumns" : [ ],                                                                                                                                                                                                          >
         |     "configuration" : {                                                                                                                                                                                                                >
         |       "delta.enableDeletionVectors" : "false"                                                                                                                                                                                          >
         |     },                                                                                                                                                                                                                                 >
         |     "createdTime" : 1733023200751                                                                                                                                                                                                      >
         |   }                                                                                                                                                                                                                                    >
         | }, {                                                                                                                                                                                                                                   >
         |   "add" : {                                                                                                                                                                                                                            >
         |     "path" : "20241201_032000_00022_p82tv_614227e8-71ab-4b9a-a8f4-f0f166fc11e0",                                                                                                                                                       >
         |     "partitionValues" : { },                                                                                                                                                                                                           >
         |     "size" : 937,                                                                                                                                                                                                                      >
         |     "modificationTime" : 1733023200743,                                                                                                                                                                                                >
         |     "dataChange" : true,                                                                                                                                                                                                               >
         |     "stats" : "{\"numRecords\":5,\"minValues\":{\"regionkey\":0,\"name\":\"AFRICA\",\"comment\":\"ges. thinly even pinto beans ca\"},\"maxValues\":{\"regionkey\":4,\"name\":\"MIDDLE EAST\",\"comment\":\"uickly special accounts cajo>
         |     "tags" : { }                                                                                                                                                                                                                       >
         |   }                                                                                                                                                                                                                                    >
         | } ]

Release notes

(x) This is not user-visible or is docs only, and no release notes are required.

@cla-bot cla-bot bot added the cla-signed label Nov 28, 2024
@github-actions github-actions bot added the delta-lake Delta Lake connector label Nov 28, 2024
@ebyhr ebyhr force-pushed the ebi/delta-history branch from 961c9f4 to a03d3f6 Compare November 28, 2024 04:01
@findinpath findinpath requested a review from mdesmet November 29, 2024 05:05
@ebyhr ebyhr force-pushed the ebi/delta-history branch from a03d3f6 to 78ef751 Compare December 1, 2024 03:20
@ebyhr ebyhr changed the title Add hidden $entries column to Delta history table Add $transactions system table to Delta Lake Dec 1, 2024
@ebyhr ebyhr force-pushed the ebi/delta-history branch from 78ef751 to c16cc73 Compare December 4, 2024 13:19
@ebyhr ebyhr force-pushed the ebi/delta-history branch from c16cc73 to d802427 Compare December 4, 2024 13:33
@ebyhr ebyhr force-pushed the ebi/delta-history branch from d802427 to f0581e4 Compare December 5, 2024 02:11
@ebyhr ebyhr requested a review from pajaks December 5, 2024 02:11
@ebyhr ebyhr merged commit 62d77f7 into trinodb:master Dec 5, 2024
22 checks passed
@ebyhr ebyhr deleted the ebi/delta-history branch December 5, 2024 03:56
@github-actions github-actions bot added this to the 467 milestone Dec 5, 2024
@mosabua
Copy link
Member

mosabua commented Dec 5, 2024

We still have to at least add it to the release notes.. and I think we also have to document it.

@mosabua
Copy link
Member

mosabua commented Dec 5, 2024

Confirmed with @martint when talking about the release notes.. if its useful for us as users and developers .. its useful for others.. so we should document.

@mosabua
Copy link
Member

mosabua commented Jan 9, 2025

@Ordinant @jhlodin @findinpath - want to collaborate to get these docs added?

@Ordinant
Copy link
Member

Ordinant commented Jan 9, 2025

@mosabua: Sure. Looks straightforward for Galaxy docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-signed delta-lake Delta Lake connector

Development

Successfully merging this pull request may close these issues.

6 participants