feat(io): Make Storage a trait
#1755
Conversation
| } | ||
|
|
||
| #[async_trait] | ||
| impl Storage for OpenDALGcsStorage { |
There was a problem hiding this comment.
It can be pretty annoying to implement nearly the same thing for all storage services, can we avoid that?
There was a problem hiding this comment.
I think this thread is relavant: https://docs.google.com/document/d/1-CEvRvb52vPTDLnzwJRBx5KLpej7oSlTu_rg0qKEGZ8/edit?disco=AAABrRO9Prk
The benefit of doing this is that users are allowed to only implement Storage for certain schemes. The annoying part of having duplicate code for multiple schemes will mostly apply to a versatile storage implementation like OpenDAL, which already has a convenient operator layer. For custom storage, I don't expect them to implement all schemes anyway (I may be wrong on this assumption)
For code duplication, I consider OpenDAL Storage to be the "managed" default storage that lives in this repo and we will have more control over the implementation. Once we have a new crate for each storage implementation(iceberg-storage-opendal), we can add some helpers to reduce the code duplication
2 participants
Which issue does this PR close?
What changes are included in this PR?
Are these changes tested?
Added some tests for the new storage builder and registry, but mostly relying on the existing tests