Iceberg materialization test

[ian-r-rose]

@thehanggit I'm reopening this with some additional experimentation around iceberg metadata. The main issue is that there can be many different versions of the iceberg metadata (in general, each change to the table results in a new version of the metadata). Snowflake keeps track of which version is the most recent, but it's not always easy to determine the correct one from a different tool trying to query the iceberg table.

This creates a new stored procedure that unloads a JSON blob to the iceberg folder in S3 that lists the correct metadata file for each iceberg table in the marts database. I was then able to use that directly in the following sample duckdb script:

-- Load the extensions allowing for S3 access and Iceberg
load iceberg;
load httpfs;

set variable name = 'IMPUTATION__DETECTOR_IMPUTED_AGG_FIVE_MINUTES';
set variable schema = 'DBT_IROSE_IMPUTATION';

-- Since this script doesn't manage the iceberg tables, it doesn't know
-- which version of the metadata is the correct one. We could guess based
-- on heuristics for file names or modification dates, but that is extremely
-- unreliable. So we unload a special file into the iceberg directory that
-- tracks the most recent version of the metadata for each iceberg table
-- (as of the time the current_table_versions.json file was written)
-- This grabs the metadata path for the table we care about.
set variable meta = (
    select metadata from read_json(
        "s3://caltrans-pems-dev-us-west-2-marts/iceberg/current_table_versions.json"
    )
    where name = getvariable('name') and schema = getvariable('schema')
);


-- Actually make a query with no authentication!
select
    station_id,
    sample_date,
    sum(volume_sum) as volume,
    count_if(
        volume_imputation_method in ('local', 'regional', 'global', 'local_avg', 'regional_avg')
    )/count(*) as pct_imputed,
from iceberg_scan(getvariable('meta'))
where sample_date > '2025-03-10' and sample_date < '2025-03-17'
group by station_id, sample_date
order by station_id, sample_date;

[thehanggit]

@thehanggit I'm reopening this with some additional experimentation around iceberg metadata. The main issue is that there can be many different versions of the iceberg metadata (in general, each change to the table results in a new version of the metadata). Snowflake keeps track of which version is the most recent, but it's not always easy to determine the correct one from a different tool trying to query the iceberg table.

This creates a new stored procedure that unloads a JSON blob to the iceberg folder in S3 that lists the correct metadata file for each iceberg table in the marts database. I was then able to use that directly in the following sample duckdb script:

-- Load the extensions allowing for S3 access and Iceberg
load iceberg;
load httpfs;

set variable name = 'IMPUTATION__DETECTOR_IMPUTED_AGG_FIVE_MINUTES';
set variable schema = 'DBT_IROSE_IMPUTATION';

-- Since this script doesn't manage the iceberg tables, it doesn't know
-- which version of the metadata is the correct one. We could guess based
-- on heuristics for file names or modification dates, but that is extremely
-- unreliable. So we unload a special file into the iceberg directory that
-- tracks the most recent version of the metadata for each iceberg table
-- (as of the time the current_table_versions.json file was written)
-- This grabs the metadata path for the table we care about.
set variable meta = (
    select metadata from read_json(
        "s3://caltrans-pems-dev-us-west-2-marts/iceberg/current_table_versions.json"
    )
    where name = getvariable('name') and schema = getvariable('schema')
);


-- Actually make a query with no authentication!
select
    station_id,
    sample_date,
    sum(volume_sum) as volume,
    count_if(
        volume_imputation_method in ('local', 'regional', 'global', 'local_avg', 'regional_avg')
    )/count(*) as pct_imputed,
from iceberg_scan(getvariable('meta'))
where sample_date > '2025-03-10' and sample_date < '2025-03-17'
group by station_id, sample_date
order by station_id, sample_date;

@ian-r-rose Thank you Ian! One question here is: does this json file stores all iceberg tables' metadata information? And we can use such one json file to find all most updated metadata information by choosing variables for the table name and schema right.

[ian-r-rose]

One question here is: does this json file stores all iceberg tables' metadata information? And we can use such one json file to find all most updated metadata information by choosing variables for the table name and schema right.

Yes, exactly!

I'm attaching an example file that was created as part of this PR (which is also hosted in S3, as you can see in the above script):

current_table_versions.json

[thehanggit]

One question here is: does this json file stores all iceberg tables' metadata information? And we can use such one json file to find all most updated metadata information by choosing variables for the table name and schema right.

Yes, exactly!

I'm attaching an example file that was created as part of this PR (which is also hosted in S3, as you can see in the above script):

current_table_versions.json

Thanks for clarification! I will follow your steps and create necessary iceberg tables for your review.