Skip to content

fix: BROS-23: Don't fail on parsing errors in storage #7567

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: develop
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 9 additions & 7 deletions label_studio/io_storages/base_models.py
Original file line number Diff line number Diff line change
Expand Up @@ -420,25 +420,27 @@
# w/o Dataflow
# pubsub.push(topic, key)
# -> GF.pull(topic, key) + env -> add_task()
logger.debug(f'Scanning key {key}')
logger.debug('Scanning key %s', key)
self.info_update_progress(last_sync_count=tasks_created, tasks_existed=tasks_existed)

# skip if key has already been synced
if n_tasks_linked := link_class.n_tasks_linked(key, self):
logger.debug(f'{self.__class__.__name__} already has {n_tasks_linked} tasks linked to {key=}')
logger.debug('%s link %s already exists with %d tasks linked', self.__class__.__name__, key, n_tasks_linked)
tasks_existed += n_tasks_linked # update progress counter
continue

logger.debug(f'{self}: found new key {key}')
logger.debug('%s: found new key %s', self.__class__.__name__, key)
try:
tasks_data = self.get_data(key)
except (UnicodeDecodeError, json.decoder.JSONDecodeError) as exc:
logger.debug(exc, exc_info=True)
raise ValueError(
f'Error loading JSON from file "{key}".\nIf you\'re trying to import non-JSON data '
f'(images, audio, text, etc.), edit storage settings and enable '
f'"Treat every bucket object as a source file"'
logger.warning(

Check warning on line 437 in label_studio/io_storages/base_models.py

View check run for this annotation

Codecov / codecov/patch

label_studio/io_storages/base_models.py#L437

Added line #L437 was not covered by tests
'Error loading JSON from file %s: %s. If you\'re trying to import non-JSON data '
'(images, audio, text, etc.), edit storage settings and enable '
'"Treat every bucket object as a source file"',
key, exc
)
continue

Check warning on line 443 in label_studio/io_storages/base_models.py

View check run for this annotation

Codecov / codecov/patch

label_studio/io_storages/base_models.py#L443

Added line #L443 was not covered by tests

if isinstance(tasks_data, dict):
tasks_data = [tasks_data]
Expand Down
Loading