Skip to content

Add eLabFTW file source from file source templates #19493

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Feb 13, 2025

Conversation

kysrpex
Copy link
Contributor

@kysrpex kysrpex commented Jan 30, 2025

Support user-defined file sources based on eLabFTWFilesSource. Add a file source template for eLabFTW and the required documentation.

Adding eLabFTW as user-defined files source

How to test the changes?

(Select all options that apply)

  • I've included appropriate automated tests.
  • This is a refactoring of components with existing test coverage.
  • Instructions for manual testing are as follows:
    1. Run eLabFTW, for example using Docker Compose (or use https://demo.elabftw.net).
    2. Enable the file source template.
    3. Create an account and generate an API Key on eLabFTW (or use the demo account from https://demo.elabftw.net).
    4. Create a new eLabFTW file source on the user preferences page, configuring the endpoint and API Key.
    5. Create an experiment or resource in eLabFTW and attach a file.
    6. Import the file to your history from the eLabFTW file source.
    7. Export your history to the eLabFTW file source.

License

  • I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

Comment on lines -172 to +193
def get_prefix(self) -> Optional[str]:
return None
def get_prefix(self, user_context: OptionalUserContext = None) -> Optional[str]:
endpoint: ParseResult = self._get_endpoint(user_context=user_context)
return self.id if self.scheme not in {"elabftw", DEFAULT_SCHEME} else (endpoint.netloc or None)
# it would make better sense to return
# `self.id if self.scheme == USER_FILE_SOURCES_SCHEME else (endpoint.netloc or None)`, where
# `USER_FILE_SOURCES_SCHEME` comes from `galaxy.managers.file_source_instances`; however, that would lead to a
# circular import (maybe `USER_FILE_SOURCES_SCHEME` should be moved to a module in a layer deeper than
# `galaxy.managers`)

def get_scheme(self) -> str:
return "elabftw"
return self.scheme if self.scheme and self.scheme != DEFAULT_SCHEME else "elabftw"
# it would make better sense to return `self.scheme if self.scheme == USER_FILE_SOURCES_SCHEME else "elabftw"`,
# but the same circular import issue as above arises
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this ended up being so weird there must be something I am not getting about the scheme... Where are schemes other than "gxfiles" and "gxuserfiles" coming from? The design intended in #19319 with a URI that only depends on the configured endpoint is not how things are supposed to work, right?

File sources reference files via a URI, while eLabFTW uses auto-incrementing positive integers. For more details read #18665 [5]. This leads to the need to declare a mapping between said identifiers and Galaxy URIs.

Those take the form elabftw://demo.elabftw.net/entity_type/entity_id attachment_id, where:

  • entity_type is either 'experiments' or 'resources'
  • entity_id is the id (an integer in string form) of an experiment or resource
  • attachment_id is the id (an integer in string form) of an attachment

For the user-defined file sources use case (when users configure one or more instances of the FilesSource via a file source template), Galaxy URIs have a different scheme and authority, taking the form gxuserfiles://file_source_id/entity_type/entity_id/attachment_id, where:

  • file_source_id is the file source identifier assigned by Galaxy

@kysrpex
Copy link
Contributor Author

kysrpex commented Jan 30, 2025

FYI @bgruening

@kysrpex kysrpex self-assigned this Jan 30, 2025
@kysrpex kysrpex added kind/enhancement kind/refactoring cleanup or refactoring of existing code, no functional changes kind/feature area/API area/backend labels Jan 30, 2025
@kysrpex kysrpex added this to the 25.0 milestone Jan 30, 2025
@kysrpex kysrpex marked this pull request as draft January 30, 2025 16:22
@kysrpex kysrpex force-pushed the elabftw_file_source_template branch from cf60ef4 to f0e4866 Compare January 30, 2025 17:02
@kysrpex kysrpex marked this pull request as ready for review January 30, 2025 17:09
@kysrpex kysrpex force-pushed the elabftw_file_source_template branch from 51063f4 to 6cdc929 Compare February 5, 2025 10:45
@kysrpex
Copy link
Contributor Author

kysrpex commented Feb 13, 2025

Thanks @davelopez for realizing that writable: true was missing (855763f).

@kysrpex kysrpex force-pushed the elabftw_file_source_template branch from a0aac1e to 855763f Compare February 13, 2025 11:04
Support user-defined file sources based on `eLabFTWFilesSource`. Add a file source template for eLabFTW and the required documentation.
Let users choose whether Galaxy should be allowed to export data to eLabFTW or they only need to import data to Galaxy.
@kysrpex kysrpex force-pushed the elabftw_file_source_template branch from c4693d4 to ea25077 Compare February 13, 2025 14:34
Copy link
Contributor

@davelopez davelopez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @kysrpex! I tested it and it works as expected now.

@davelopez
Copy link
Contributor

Test failures seem unrelated

@davelopez davelopez merged commit 64d385f into galaxyproject:dev Feb 13, 2025
51 of 56 checks passed
@kysrpex kysrpex deleted the elabftw_file_source_template branch February 17, 2025 09:22
@ahmedhamidawan ahmedhamidawan changed the title eLabFTW file source from file source templates Add eLabFTW file source from file source templates May 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/API area/backend kind/bug kind/enhancement kind/feature kind/refactoring cleanup or refactoring of existing code, no functional changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants