File DataType Roadmap

[universalmind303]

The new daft.File datatype will provide first-class support for handling file data across local and remote storage, enabling seamless file operations in distributed environments.

Milestones

• Initial MVP with basic local & memory support
• Object store support (S3, HTTP, etc.)
• Buffered reading for local & remote files
• User-configurable chunk sizes via daft.context
• Metadata access expressions (size, modified date, etc.)
• Tempfile integration
• Writeable file support
• new table function (daft.from_files? ) to return a single column of File datatype
  • This will eventually replace the existing daft.from_glob_paths

Architecture

daft.File serves as the abstract interface users interact with, implemented by concrete classes:

PathFile: handles files stored anywhere with a path (local or remote)

MemoryFile: for in-memory byte arrays

This separation exists because Python file APIs behave differently for filesystem-backed files versus in-memory data (FileIO vs BytesIO). Each Python class implements specific dunder methods to maintain compatibility with standard Python file interfaces.

While the Python classes provide the interface, the actual implementation lives in Rust-based PyDaftFile, which maintains optimized backends for different storage types:

• Local filesystem access
• Remote object stores with buffered reading
• In-memory byte access

This architecture allows us to implement storage-specific optimizations (like network buffering for S3 or HTTP) while presenting a consistent interface.

Key Design Principles

1. daft.File works both within dataframes and as standalone objects
2. Optimized backend readers for different sources (buffered network access, etc.)
3. Consistent API regardless of storage location

Example Usage

# Basic usage
file = daft.File("s3://bucket/image.jpg", io_config)
content = file.read()

# UDF usage 
@daft.func
def read_from_file(file: daft.File) -> bytes:
    return file.read()

df = df.select(read_from_file(df["file"]))

# expression methods
df = df.select(
    df["file"].size().alias('file_size'),
    df["file"].last_modified().alias('modified_time')
)

# Temporary file access
with remote_file.to_tempfile() as temp_path:
    subprocess.run(["ffmpeg", "-i", temp_path, "output.jpg"])