Add data retention policy

[asmacdo]

Heres a sketch of a possible data retention policy. Lets iron out what we want here prior to implementation.

Fixes: #182

from Yarik's initial thoughts : #177 (comment)

[yarikoptic]

I think it is good for the starting point. After implemented/deployed we will see how it could be improved

[yarikoptic]

In case user is still active -- I think it would be useful to report to the long running users, after reaching some threshold on any of those folders (e.g. 50MB) asking to clean them up.

[kabilar]

Hi @asmacdo, should we add a separate point here about monitoring and reporting the quotas of cache directories for active users?

[yarikoptic]

meanwhile it might be worth creating a simple data record schema to store those records as well so they could be reused by the tools to assemble higher level stats etc.

[kabilar]

Thank you, @asmacdo. This is great. A few suggestions are listed above.

[kabilar]

Hi @asmacdo, please let me know when this is ready for review. And then we can update the DANDI Terms and Policies as needed.

[kabilar]

@asmacdo Continuing the discussion from Slack.

As we work to ephemeral environments and given our current strategy of notifying users monthly, perhaps we should just have a policy that users with data totaling more than 10 GB would get an email notice?

Proposed updated email template:

Hi <github/dandi username>,

The DANDI team is working to reduce our DANDI Hub costs.  A large portion of our costs include data stored on [DANDI Hub](https://hub.dandiarchive.org/) (not DANDI Archive).

There is currently about X GB stored under your user directory on DANDI Hub.

The data storage available on the DANDI Hub is meant for environment management and should not exceed 10GB.  Data files should be uploaded to DANDI Archive.  Please email [email protected] if you need to store more than 10 GB.  We will review each request individually and work with you to find a solution for your compute requirements.

Can you please review your files stored on DANDI Hub, upload any relevant files to your respective Dandisets on DANDI Archive, and delete any unused files on DANDI Hub?

Thank you.

DANDI Team

[kabilar]

I will provide some suggestions based on the recent standup meeting where we decided to reset home directories after 45 days of not logging in to JuptyerHub.

[kabilar]

Hi @asmacdo, I have simplified the policy based on our team meeting this week. Please review suggestions and once you are ready we will need to update the DANDI Terms and Policies as needed.

[kabilar]

This policy is inline with the General Policies v1.1.0 listed here for DANDI Hub. So we will not need to update the DANDI Terms and Policies.

[kabilar]

Hi @asmacdo, just one more suggestion and then we can merge this document.

cc @satra @yarikoptic @bendichter

[bendichter]

lgtm