Skip to content

MCR-3126 OCFL filesystem #2175

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 54 commits into
base: main
Choose a base branch
from
Open

MCR-3126 OCFL filesystem #2175

wants to merge 54 commits into from
+12,250 −166

Conversation

Mewel
Copy link
Member

@Mewel Mewel commented Jun 10, 2024

@Mewel Mewel requested review from yagee-de and sebhofmann June 10, 2024 14:35
@Mewel Mewel changed the title MCR-3126 ocfl filesystem draft MCR-3126 ocfl filesystem Jun 10, 2024
@Mewel Mewel changed the title MCR-3126 ocfl filesystem MCR-3126 OCFL filesystem Jun 10, 2024
@Mewel Mewel force-pushed the issues/MCR-3126-OCFL-FileSystem branch 2 times, most recently from 4ee8872 to a54ef34 Compare July 4, 2024 14:22
@Mewel Mewel force-pushed the issues/MCR-3126-OCFL-FileSystem branch from 6f91842 to d6c8495 Compare September 3, 2024 13:20
@Mewel Mewel force-pushed the issues/MCR-3126-OCFL-FileSystem branch 2 times, most recently from 4f561ce to cf2147e Compare October 29, 2024 11:58
@Mewel Mewel force-pushed the issues/MCR-3126-OCFL-FileSystem branch from 3f5f323 to e1eef38 Compare October 30, 2024 15:51
@Mewel Mewel force-pushed the issues/MCR-3126-OCFL-FileSystem branch from e7ac1c3 to bff4458 Compare January 14, 2025 13:19
@Mewel Mewel force-pushed the issues/MCR-3126-OCFL-FileSystem branch 2 times, most recently from 7ce5ab9 to ce3299e Compare February 24, 2025 12:36
@Mewel Mewel force-pushed the issues/MCR-3126-OCFL-FileSystem branch from 23eb8ee to 0480f8a Compare February 25, 2025 15:42
@Mewel Mewel force-pushed the issues/MCR-3126-OCFL-FileSystem branch 2 times, most recently from 6fb9cca to 10a4deb Compare March 20, 2025 18:25
@Mewel Mewel marked this pull request as ready for review March 20, 2025 18:25
@toKrause toKrause requested review from rsteph-de and removed request for sebhofmann March 25, 2025 14:19
@Mewel Mewel force-pushed the issues/MCR-3126-OCFL-FileSystem branch 3 times, most recently from c80414f to c963e3b Compare April 8, 2025 15:52
@Mewel Mewel force-pushed the issues/MCR-3126-OCFL-FileSystem branch from 158db82 to 1759378 Compare April 14, 2025 11:23
@Mewel Mewel force-pushed the issues/MCR-3126-OCFL-FileSystem branch from 43b7244 to 810e4e4 Compare May 6, 2025 12:29
@sebhofmann
Copy link
Member

sebhofmann commented May 9, 2025 •
edited
Loading

I found a conceptional Problem, which leads to errors in the tiling process. The MCRTreeCopier commits and restarts database transactions. The tiling queue runs in another thread is now able to find the commited tiling jobs and now starts tiling. The ocfl transaction of the Thread in with the MCRTreeCopier did not yet commit its OCFL Transaction, so the files which should be tiled do not exist yet.
We introduced Transaction commits in the MCRTreeCopier, so that in a large upload the connection to the database is released. In my opinion the cleanest aproach would be to remove the transaction restarts from the MCRTreeCopier, to always keep the file transaction and the database transaction in sync and also prevent weird rollback errors.

Meanwhile the uploads happen in another transaction and are stored in a local directory, so the "long" transaction only runs long to upload from the mycore machine to s3, or to copy from one local to another local directory, which should not take that long.

@Mewel Mewel force-pushed the issues/MCR-3126-OCFL-FileSystem branch 2 times, most recently from 360ce0b to f7d26f4 Compare May 14, 2025 11:45
@Mewel Mewel force-pushed the issues/MCR-3126-OCFL-FileSystem branch from 7eac844 to 947008b Compare June 23, 2025 07:48
@Mewel Mewel force-pushed the issues/MCR-3126-OCFL-FileSystem branch from 2603b9e to f0fd713 Compare July 9, 2025 10:10
@yagee-de yagee-de requested a review from Copilot July 16, 2025 07:11
Mewel and others added 26 commits July 17, 2025 12:31
@Mewel
MCR-3126 fix aws pom problems
15b3b86
@Mewel
MCR-3126 introduce remote filestore for OCFL
3846093
@Mewel
MCR-3126 add 'soft' delete support
a41e68d
@Mewel
MCR-3126 rm unecessary remote file store
07d6f56
@Mewel
MCR-3126 improve file tracker with better rename support
be8a761
@Mewel
MCR-3126 fix bug because of API changes
2ff5fce
@Mewel
MCR-3126 improve seekable chaching channel
b8b138e
@Mewel
MCR-3126 fix import order
20ebd9d
@Mewel
MCR-3126 use junit5
2b8cee5
@Mewel
MCR-3126 rm junit5 vintage engine & fix assertThrows
ca052b0
@Mewel
MCR-3126 fix pom.xml order
452d5dc
@Mewel
MCR-3126 fix pom.xml order
4a50765
@Mewel
MCR-3126 fix codacy issues
eb78d2c
@Mewel
simplify fileKey and probeContent
7289f59
@Mewel
MCR-3126 fix MCROCFLCachingSeekableByteChannel S3 buffer bug
a1099b1
@Mewel
MCR-3126 improve ocfl
8b8b96d 
* add restore derivate command
* add describe object command
* fix MCRVersionedPath#getParent does not return a versioned path
@Mewel
MCR-3126 fix rolling storage was not tracked properly
4c73024
@Mewel
MCR-3126 rename some storage classes to be more precise
b84eebf
@Mewel
MCR-3126 prepare for rewriting rolling cache
ac1dc9e
@Mewel
MCR-3126 improve ocfl
827becc 
- simplify virtual objects
- rewrote remote storage to use digest instead of path
- update junit test
- improve javadoc
@Mewel
MCR-3126 fix javadoc
783d33f
@Mewel
MCR-3126 fix pom.xml
d913917
@Mewel
MCR-3126 support attribute view in FileStore
e7a63c6
@Mewel
MCR-3126 improve remote storage
723da7e 
- cp from transactional storage to remote storage after commit
- use "real" CAS files in remote storage
@Mewel
MCR-3126 fix cp to remote storage
08b3d90 
the first commit to a derivate was not stored on the remote storage due to wrongful calculation of the version path
@Mewel
MCR-3126 stream directly to target
52cd99e 
in remote repos stream directly to target path when makeing a
copy/externalCopy instead of streaming the source first and then copy
from there
@Mewel Mewel force-pushed the issues/MCR-3126-OCFL-FileSystem branch from dba45aa to 52cd99e Compare July 17, 2025 10:33
Mewel added 3 commits July 17, 2025 12:50
@Mewel
MCR-3126 use new normalize() for digest in journal
79f8cb7
@Mewel
MCR-3126 simplify remote storage journal
235df7a
@Mewel
MCR-3126 rm TOUCH from journal
59c7a83 
- makes the journal smaller
- compact every hour to store LRU queue
- memory is enough to store "touch" position
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

Copilot code review Copilot Copilot left review comments

@yagee-de yagee-de Awaiting requested review from yagee-de

@rsteph-de rsteph-de Awaiting requested review from rsteph-de

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Mewel @sebhofmann