Skip to content

Conversation

GeigerJ2
Copy link
Contributor

@GeigerJ2 GeigerJ2 commented Oct 6, 2025

For (possibly) time-intensive preparatory tasks during the dumping (querying from db, filtering based on certain criteria such as previously being dumped, or exclusion based on filters, etc.), additional logging is added before and after the operation, while the operations themselves are wrapped in progress bars, where applicable. This makes the output when running the command much more user-friendly, as it doesn't seem like the code is just stuck during a more time-intensive operation, but the display remains responsive and shows what's going on internally.

This is not a fix to #6913, per se, in the same way that what was reported in #6913 was not an actual bug. Possible speed improvements of the dumping-related code are a possible consideration for v2.8.0. For the a v2.7.2 patch release, I deem the additional logging in this PR to be sufficient.

  • TODO: drop truncated text for progress bars where report message shown above is informative (thanks, @Minotakm)

@GeigerJ2 GeigerJ2 force-pushed the dump-logging-when-time-intensive branch from 40e616e to b876283 Compare October 6, 2025 12:50
Copy link

codecov bot commented Oct 6, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.28%. Comparing base (cfbbd68) to head (87058a9).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #7046      +/-   ##
==========================================
- Coverage   79.29%   79.28%   -0.00%     
==========================================
  Files         566      566              
  Lines       43847    43847              
==========================================
- Hits        34763    34761       -2     
- Misses       9084     9086       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@GeigerJ2 GeigerJ2 marked this pull request as ready for review October 6, 2025 13:20
@GeigerJ2 GeigerJ2 requested a review from agoscinski October 6, 2025 13:20
@GeigerJ2 GeigerJ2 changed the title Additional logging for group-node mapping creation during dump operation Additional logging during dump operations Oct 6, 2025
@GeigerJ2
Copy link
Contributor Author

GeigerJ2 commented Oct 6, 2025

@agoscinski, nice, i'm running into sql errors when running dump with the testing group 🥲

❯ verdi profile dump --groups mygroup
Warning: This is a new feature which is still in its testing phase. If you encounter unexpected behavior or bugs, please report them via Discourse or GitHub.
Report: No output path specified. Using default: '/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/dump-fixes'
Report: Building group-node mapping for 1 specified groups...
Report: Querying node memberships for 1 groups...
Report: Retrieving group-node relationships from database...
Report: Processing 11865 group-node relationships...
Report: Completed group-node mapping.
Report: Starting dump of profile `dump-fixes` in incremental mode...
Report: Detecting changes since last dump. This may take a while for large databases...
Report: Detecting node changes...
Report: Querying calculation nodes from database...
Report: Retrieved 1300 calculation nodes.
Report: Applied relevant filters to calculations.
Report: Querying workflow nodes from database...
Report: Retrieved 1285 workflow nodes.
Report: Applied relevant filters to workflows.
Report: Detected 2585 new/modified nodes and 0 deleted nodes.
Report: Detecting group changes...
Report: Detected 1 new, 0 modified, and 0 deleted groups.
Report: Processing group changes...
Report: Dumping 2585 nodes for group 'mygroup'
Critical: Unexpected error during dump of dump-fixes:
 ((sqlite3.DatabaseError) database disk image is malformed
[SQL: SELECT db_dbnode.id AS db_dbnode_id, db_dbnode.uuid AS db_dbnode_uuid, db_dbnode.node_type AS db_dbnode_node_type, db_dbnode.process_type AS db_dbnode_process_type, db_dbnode.label AS db_dbnode_label, db_dbnode.description AS db_dbnode_description, db_dbnode.ctime AS db_dbnode_ctime, db_dbnode.mtime AS db_dbnode_mtime, db_dbnode.attributes AS db_dbnode_attributes, db_dbnode.extras AS db_dbnode_extras, db_dbnode.repository_metadata AS db_dbnode_repository_metadata, db_dbnode.dbcomputer_id AS db_dbnode_dbcomputer_id, db_dbnode.user_id AS db_dbnode_user_id
FROM db_dbnode
WHERE db_dbnode.id = ?]
[parameters: (7329,)]
(Background on this error at: https://sqlalche.me/e/20/4xp6)).
Traceback (most recent call last):
  File "/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/src/aiida/tools/_dumping/executors/process.py", line 113, in dump
  File "/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/src/aiida/tools/_dumping/executors/process.py", line 322, in _execute_dump_duplicate
  File "/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/src/aiida/tools/_dumping/executors/process.py", line 343, in _dump_node_content
  File "/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/src/aiida/tools/_dumping/executors/process.py", line 508, in _dump_calculation_content
  File "/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/src/aiida/tools/_dumping/executors/process.py", line 553, in _dump_calculation_io_files
  File "/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/src/aiida/orm/nodes/repository.py", line 361, in copy_tree
  File "/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/src/aiida/repository/repository.py", line 518, in copy_tree
  File "/usr/lib/python3.10/pathlib.py", line 1175, in mkdir
OSError: [Errno 30] Read-only file system: '/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/dump-fixes/groups/mygroup/workflows/HpBaseWorkChain-7329/01-iteration_01-HpCalculation-7439/node_inputs/hubbard_structure'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1963, in _exec_single_context
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 943, in do_execute
sqlite3.DatabaseError: database disk image is malformed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/src/aiida/tools/_dumping/executors/process.py", line 104, in dump
  File "/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/src/aiida/tools/_dumping/executors/process.py", line 238, in _execute_dump_primary
  File "/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/src/aiida/tools/_dumping/executors/process.py", line 345, in _dump_node_content
  File "/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/src/aiida/tools/_dumping/executors/process.py", line 604, in _dump_children
  File "/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/src/aiida/tools/_dumping/executors/process.py", line 115, in dump
  File "/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/src/aiida/tools/_dumping/executors/process.py", line 129, in _handle_dump_error
  File "/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/src/aiida/orm/entities.py", line 378, in pk
  File "/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/src/aiida/storage/psql_dos/orm/entities.py", line 74, in id
  File "/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/src/aiida/storage/psql_dos/orm/utils.py", line 81, in __getattr__
  File "/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/src/aiida/storage/psql_dos/orm/utils.py", line 107, in is_saved
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 566, in __get__
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 1086, in get
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 1116, in _fire_loader_callables
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/orm/state.py", line 803, in _load_expired
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/orm/loading.py", line 1670, in load_scalar_attributes
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/orm/loading.py", line 509, in load_on_ident
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/orm/loading.py", line 694, in load_on_pk_identity
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2365, in execute
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2251, in _execute_internal
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1415, in execute
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 523, in _execute_on_connection
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1637, in _execute_clauseelement
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1842, in _execute_context
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1982, in _exec_single_context
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2351, in _handle_dbapi_exception
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1963, in _exec_single_context
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 943, in do_execute
sqlalchemy.exc.DatabaseError: (sqlite3.DatabaseError) database disk image is malformed
[SQL: SELECT db_dbnode.id AS db_dbnode_id, db_dbnode.uuid AS db_dbnode_uuid, db_dbnode.node_type AS db_dbnode_node_type, db_dbnode.process_type AS db_dbnode_process_type, db_dbnode.label AS db_dbnode_label, db_dbnode.description AS db_dbnode_description, db_dbnode.ctime AS db_dbnode_ctime, db_dbnode.mtime AS db_dbnode_mtime, db_dbnode.attributes AS db_dbnode_attributes, db_dbnode.extras AS db_dbnode_extras, db_dbnode.repository_metadata AS db_dbnode_repository_metadata, db_dbnode.dbcomputer_id AS db_dbnode_dbcomputer_id, db_dbnode.user_id AS db_dbnode_user_id
FROM db_dbnode
WHERE db_dbnode.id = ?]
[parameters: (7439,)]
(Background on this error at: https://sqlalche.me/e/20/4xp6)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1963, in _exec_single_context
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 943, in do_execute
sqlite3.DatabaseError: database disk image is malformed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/src/aiida/cmdline/commands/cmd_profile.py", line 384, in profile_dump
  File "/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/src/aiida/manage/configuration/profile.py", line 387, in dump
  File "/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/src/aiida/tools/_dumping/engine.py", line 127, in dump
  File "/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/src/aiida/tools/_dumping/engine.py", line 277, in _dump_profile
  File "/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/src/aiida/tools/_dumping/executors/profile.py", line 80, in dump
  File "/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/src/aiida/tools/_dumping/executors/collection.py", line 193, in _process_group
  File "/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/src/aiida/tools/_dumping/executors/collection.py", line 160, in _dump_nodes
  File "/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/src/aiida/tools/_dumping/executors/process.py", line 115, in dump
  File "/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/src/aiida/tools/_dumping/executors/process.py", line 129, in _handle_dump_error
  File "/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/src/aiida/orm/entities.py", line 378, in pk
  File "/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/src/aiida/storage/psql_dos/orm/entities.py", line 74, in id
  File "/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/src/aiida/storage/psql_dos/orm/utils.py", line 81, in __getattr__
  File "/home/geiger_j/aiida_projects/aiida-dev/git-repos/aiida-core/src/aiida/storage/psql_dos/orm/utils.py", line 107, in is_saved
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 566, in __get__
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 1086, in get
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 1116, in _fire_loader_callables
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/orm/state.py", line 803, in _load_expired
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/orm/loading.py", line 1670, in load_scalar_attributes
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/orm/loading.py", line 509, in load_on_ident
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/orm/loading.py", line 694, in load_on_pk_identity
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2365, in execute
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2251, in _execute_internal
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1415, in execute
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 523, in _execute_on_connection
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1637, in _execute_clauseelement
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1842, in _execute_context
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1982, in _exec_single_context
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2351, in _handle_dbapi_exception
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1963, in _exec_single_context
  File "/home/geiger_j/.aiida_venvs/aiida-dev/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 943, in do_execute
sqlalchemy.exc.DatabaseError: (sqlite3.DatabaseError) database disk image is malformed
[SQL: SELECT db_dbnode.id AS db_dbnode_id, db_dbnode.uuid AS db_dbnode_uuid, db_dbnode.node_type AS db_dbnode_node_type, db_dbnode.process_type AS db_dbnode_process_type, db_dbnode.label AS db_dbnode_label, db_dbnode.description AS db_dbnode_description, db_dbnode.ctime AS db_dbnode_ctime, db_dbnode.mtime AS db_dbnode_mtime, db_dbnode.attributes AS db_dbnode_attributes, db_dbnode.extras AS db_dbnode_extras, db_dbnode.repository_metadata AS db_dbnode_repository_metadata, db_dbnode.dbcomputer_id AS db_dbnode_dbcomputer_id, db_dbnode.user_id AS db_dbnode_user_id
FROM db_dbnode
WHERE db_dbnode.id = ?]
[parameters: (7329,)]
(Background on this error at: https://sqlalche.me/e/20/4xp6)

fish: Job 1, 'verdi profile dump --groups myg…' terminated by signal SIGBUS (Misaligned address error)
exec: Failed to execute process '/home/geiger_j/.cargo/bin/atuin', unknown error number 5
disown: There are no suitable jobs

~/aiida_projects/aiida-dev/git-repos/aiida-core dump-logging-when-time-… *4                                                                                                                                                                         󰌠 3.10.12 (aiida-dev) 15:19:50
❯ exec: Failed to execute process '/usr/bin/fish', unknown error number 5

~/aiida_projects/aiida-dev/git-repos/aiida-core dump-logging-when-time-… *4                                                                                                                                                                         󰌠 3.10.12 (aiida-dev) 15:19:50
❯

@GeigerJ2 GeigerJ2 force-pushed the dump-logging-when-time-intensive branch from a00fa52 to 6d5469c Compare October 15, 2025 10:58
@GeigerJ2 GeigerJ2 moved this to In progress in aiida-core v2.7.2 Oct 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In progress

Development

Successfully merging this pull request may close these issues.

verdi profile dump --groups <GROUP> does not work while verdi group dump <GROUP> works

2 participants