Performance and Memory Issues with Large File Preview Downloads

[clostao]

Problem

The current file preview implementation downloads the entire file before showing the preview. This approach can cause several issues with large files:

1. User Experience:

   • Users must wait for the complete file download before seeing any preview
   • No progress indication during download

2. Memory Usage:

   • The file is loaded entirely into memory as a Blob
   • For encrypted files, an additional ArrayBuffer copy is created for decryption
   • For text files, a third copy is created when converting to text
   • This can lead to browser crashes or poor performance with large files

Current Implementation

// Current approach in FilePreview/index.tsx
const response = await fetch(gatewayUrl);
const blob = await response.blob();
// For encrypted files:
const encryptedFileData = {
  dataArrayBuffer: await blob.arrayBuffer(),
  // ...
};

Proposed Solutions

For UX

1. Progressive Loading:
   • Show loading progress during download
   • Enable partial preview of downloaded portions
   • Particularly important for text and PDF files

For Memory Usage

1. Streaming Preview with Range Requests:

   • Requires gateway support for range requests
   • Show preview as chunks become available (only feasible for some type of file e.g json)
   • Allow cancellation of remaining download if user closes preview (using range requests)

2. Preview Size Limits:

   • Add configurable maximum preview size (we must add this one since the file ultimately will be stored in memory when previewed so 1TB files would crash in any scenario)
   • Warn users when files exceed preview limit and offer direct download instead of preview for large files

Technical Considerations

• Gateway needs to support HTTP Range requests
• Need to modify encryption handling to work with chunks
• Preview component needs to handle partial/streaming content
• Consider adding file size checks before attempting preview

Related Components

• FilePreview/index.tsx
• @autonomys/auto-design-system/FilePreview
• Gateway download endpoint

[clostao]

@jfrank-summit For an initial solution, I'd go for:

1. Max preview size limit (50MB)

   • Show metadata + download button for larger files
   • Prevents browser memory issues

2. Download progress indicator

   • Use fetch with ReadableStream to track progress
   • Gives users feedback during download

3. Max size for being retrieved from the DSN (1MB)

   • Prevents for cloudfare timeouts
   • Gives a better context of the state of the file to the user