Skip to content

Added GCS uploads to additional outputs #49

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Added GCS uploads to additional outputs #49

wants to merge 1 commit into from

Conversation

robsv
Copy link
Contributor

@robsv robsv commented May 12, 2025

Added Google Cloud Storage uploads for outputs from the following sources:
inputs/annotations.py
inputs/dvidseg.py
inputs/sizes.py
outputs/neurotransmitters.py
outputs/reports.py
This does not include HTML or PNG files (yet).

Test uploads are viewable at https://console.cloud.google.com/storage/browser/flyem-snapshots/2023-09-26-932d3f

cc @stuarteberg

@robsv robsv requested a review from stuarteberg May 12, 2025 16:41
@robsv robsv self-assigned this May 12, 2025
@robsv robsv added the enhancement New feature or request label May 12, 2025
@stuarteberg
Copy link
Member

This changeset involves edits to many files and adds the same config setting in several parts of the config. Also, this highlights the fact that feather files are being exported during supposedly "input" steps.

This all reveals a problem with the way my original code was factored. I think we need to fix up the original logic first. Perhaps all of the exports (feather files and GCS upload) should probably occur during the 'flat connectome' export step. (I would exclude certain feather files which are meant more as debugging outputs. I would also exclude the feather files that are part of the 'reports' step.)

Side note: Before we embark on any refactoring of the code, I have changes in my local development branch I should merge...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants