Skip to content

SNOW-2409156: Add write_parquet function equivalent to write_arrow #3888

@nicornk

Description

@nicornk

What is the current behavior?

There is no native function in snowpark-python or the snowflake-python-connector to ingest a folder of parquet files in a performant way to Snowflake. The closest is Session.write_arrow which already has all required logic.

What is the desired behavior?

Add Session.write_parquet(folder_with_parquet_files, ...) that reuses the logic from write_arrow. Refactor write_arrow to first create the parquet files and than calls Session.write_parquet

How would this improve snowflake-snowpark-python?

Many customers already have parquet files written by other engines that need to be quickly ingested into Snowflake. Avoiding the arrow step adds the most performant ingestion method to snowpark-python.

References, Other Background

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions