Skip to content

[TASK] Refined file downloads should use AWS S3 #221

@jakewheeler

Description

@jakewheeler

🚧 What needs to be done

The refiner UI allows users to upload an eCR and then download the refined files that are produced. However, there are several issues with the current implementation that need to be addressed so that this feature can continue to exist and be usable in a production environment.

🤔 Why it needs to be done

In order to make these feature resilient and reliable, we should store the generated files in AWS S3. This will allow us to:

  1. Create an S3 lifecycle policy to delete files after a set period of time
  2. Encrypt the files
  3. Provide signed download links back to the user, which will only allow the uploader to access the file
  4. Allow containers to scale if needed without breaking this functionality (files are currently tracked in-memory)

🏗️ Acceptance Criteria

  • Application performs required checks prior to file upload
  • Simulate S3 locally as part of the refiner's docker-compose.yaml setup (take a look at localstack)
  • Application should create a refined zip and store them in S3
  • Upload uploading, the application should produce a signed S3 link that can be provided to the user for download
  • We can likely remove the /download route from demo.py since the returned download link will be a signed S3 link
  • Remove code from demo.py and elsewhere related to handling file cleanup

⚙️ Technical Notes

Localstack should spin up with the rest of the docker compose infrastructure. We may need some S3 related environment variables added to the application.

📜 Additional context

N/A

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions