NotImplementedError: Importing data to aiida-s3 isn't supported

[sebaB003]

I was trying to import an exported archive but I found out that it isn't implemented in aiida-s3.
The ability to import data from the standard object storage to aiida-s3 and vice-versa is a must have to allow the data sharing between multiple types of deployments.

Is there any plan to implement this feature, otherwise can you teach us how to do it, or point us to to useful resources?

Thank you!

[rikigigi]

Hello,
is this a feature that should be implemented partially in aiida-core also? I see that the error originates from here:
https://github.com/aiidateam/aiida-core/blob/98ffc331d154cc7717861fde6c908ec510003926/src/aiida/tools/archive/imports.py#L1161

that means that to solve the issue it is necessary to implement an hash -> uuid translation mechanism?

is it better to try to generalize the aiida-core routine or to build a custom command in aiida-s3 to do the import/export?

thank you

[sphuber]

The problem is indeed that the archiving implementation in aiida-core currently assumes that the repository implementations use the same format for the object keys. Since the archive is a special case of the StorageBackend, it is just another storage and the archiving mechanism uses this to simply stream data from one storage to another, but in doing so requires the repositories keys are compatible. In the archive storage, the keys are the SHA of the object themselves, but for aiida-s3 the key is just a UUID4. Not sure how difficult it will be to generalize the archiving implementation in aiida-core that would allow aiida-s3 backends to also be used. I am now working in industry though and it is unlikely I will have the time for it anytime soon.

[rikigigi]

Thank you! So we will work on this. We already did a proof-of-concept that is able to import archives generated with a standard aiida installation into a deployment done with aiida-s3. We need to work a bit on the aiida-core side, so we will start a discussion there

reference to the proof-of-concept commits:

rikigigi/aiida-core@bf04cb8
rikigigi/aiida-core@4547019

rikigigi@4870c06