v0.4.0 - Docs, docs, docs 📖

The v0.4.0 release is dominated by the complete overhaul of the user documentation (check it out at https://lakefs-spec.org/) 🚀

The documentation now features a quickstart guide, a full user guide, and an improved tutorial on how to use lakeFS-spec in a data science project. Feedback on and contributions to the documentation are highly appreciated!

🚨 Breaking Changes

• The LakeFSFileSystem.scope context manager has been removed from the API in #175, since its functionality has been superseded

What's Changed

• Make pyyaml dependency mandatory by @AdrianoKF in #143
• Restructure README [ENG-122] by @AdrianoKF in #129
• Run pre-commit autoupdate, delete unnecessary gitkeep file by @nicholasjng in #144
• Remove file upload performance warning by @nicholasjng in #146
• Improve lakeFS transactions by using deque, placeholders for commit SHAs by @nicholasjng in #140
• Move demo lakectl to tutorials folder by @maxmynter in #148
• Integrate .ipynb to CI and move to docs by @maxmynter in #119
• Quickstart guide [ENG-146] by @AdrianoKF in #142
• Delete stale black config and comments, fix error handling in blockstore puts by @nicholasjng in #150
• Add lakeFS file system configuration doc by @nicholasjng in #151
• Reword Python environment installation guide by @nicholasjng in #152
• Add exist_ok flag to all resource-creating client helpers by @nicholasjng in #154
• Docs: Landing page [ENG-147] by @AdrianoKF in #155
• Docs: Branding fixes by @AdrianoKF in #157
• Improvements to ls implementation by @AdrianoKF in #166
• Add file system usage documentation [ENG-148] by @nicholasjng in #156
• Docs: Add third-party integrations user guide by @AdrianoKF in #167
• Allow leading lakefs:// scheme in input paths by @nicholasjng in #169
• Change fs.info() logic to recognize non-slash-terminated directories by @nicholasjng in #160
• Add transaction usage document [ENG-150] by @nicholasjng in #171
• Documentation cleanup [ENG-198] by @AdrianoKF in #170
• Docs: Post-hoc cleanup of third-party integrations user guide [ENG-149] by @AdrianoKF in #172
• Change certain aspects of the data science tutorial notebook by @nicholasjng in #158
• Add doc-strings to the client_helpers by @maxmynter in #168
• Add HTTP409 check in create_repository for forward compatibility by @nicholasjng in #177
• Docs: Render pre-release banner for main by @AdrianoKF in #176
• Adapt create_tag client helper to reflect tag immutability by @maxmynter in #179
• Remove _strip_protocol overload and usages by @nicholasjng in #174
• Remove LakeFSFileSystem.scope context manager by @nicholasjng in #175
• Add depagination helper by @nicholasjng in #180

Full Changelog: v0.3.0...v0.4.0

v0.3.0 - Transaction support

This release introduces transactions as the new way of carrying out versioning operations during file transfers. They supersede hooks, which were removed from the package.

Type hints were added to all public lakefs-spec interfaces, as well as fsspec callback support to file transfers.

This release also marks the transition to the lakefs-sdk package as the way of interfacing with the lakeFS server from Python. As such, older lakeFS server versions might not be supported anymore.

What's Changed

• Fix unbound local error by moving up definition of exists boolean by @nicholasjng in #111
• Change lakeFS package to lakefs-sdk in place of lakefs-client by @nicholasjng in #107
• Add README on hack/ folder resources by @nicholasjng in #115
• Implement Demo Feedback from [ENG-111] by @maxmynter in #117
• Run pre-commit autoupdate by @nicholasjng in #118
• Add fsspec callback support for put_file and get_file by @nicholasjng in #120
• Drop isort hook in favor of ruff plugin by @nicholasjng in #122
• Remove stale lakeFS SDK branch statement by @nicholasjng in #124
• Minor test cleanups by @nicholasjng in #125
• Add type hints to lakeFS file and file system [ENG-123] by @nicholasjng in #123
• Keep only single development docs version for main branch by @AdrianoKF in #131
• Upgrade to mike to v2.0.0 by @AdrianoKF in #132
• Switch out black for ruff format by @nicholasjng in #134
• Explicitly check for existence in LakectlConfig.read() by @nicholasjng in #133
• Add ruff-format hook by @nicholasjng in #136
• Add lakeFS transactions by @nicholasjng in #135
• Defer to super class for get_file by @nicholasjng in #137
• Update demo to use transactions, bump lakefs-spec to HEAD and urllib by @nicholasjng in #138
• Remove hooks functionality by @nicholasjng in #139

Full Changelog: v0.2.1...v0.3.0

v0.2.1 - User docs

This release adds bare-bones user documentation which will be expanded in subsequent releases.

What's Changed

• Move ensure_branch out of warn_on[...] check by @maxmynter in #108
• Update demo .ipynb to current version and include rev_parse [ENG-99] by @maxmynter in #109
• Create bare-bones user documentation [ENG-110] by @AdrianoKF in #110

Full Changelog: v0.2.0...v0.2.1

v0.2.0 - Presigned URL blockstore upload, Python 3.9 compatibility

What's Changed

• Add Reproducibility Example to Readme.md by @maxmynter in #93
• Run hooks even if precheck skips fs action [ENG-100] by @maxmynter in #100
• Remove no-context host specifications in FS constructor, drop `yaml… by @nicholasjng in #101
• Allow LakeFS files to decay standard Blockstore files on request [ENG-92] by @maxmynter in #95
• Small fixes re: blockstore puts by @nicholasjng in #102
• Remove setuptools/poetry entrypoint registration by @nicholasjng in #104
• Introduce future annotations and guard imports for Python 3.9 compat by @nicholasjng in #105
• Add badges to README by @AdrianoKF in #106

Full Changelog: v0.1.6...v0.2.0

v0.1.6

This release adds the ability to copy files in a repo by implementing LakeFSFileSystem.cp_file, as well as a comprehensive demo to explore the features of lakefs-spec.

The client helpers module was greatly expanded by @maxmynter.

Minor bug fixes include removal of stdout handlers from all loggers.

What's Changed

• Update lakeFS image in local tests and CI, bump ruff by @nicholasjng in #87
• Add file upload and download events and corresponding LakeFSFile hooks by @nicholasjng in #85
• Implement LakeFSFilesystem.cp_file by @nicholasjng in #84
• Implement directory listings caching for ls by @nicholasjng in #86
• Add new client_helper functions by @maxmynter in #88
• Add rev-parse client helper by @maxmynter in #90
• Remove stdout handlers from module loggers by @nicholasjng in #91
• Create a Demo for LakeFS-spec and Using LakeFS in a small scale e-2-e example project by @maxmynter in #67
• [ENG-96] Run pre-commit autoupdate by @nicholasjng in #97
• Update Python development dependencies and lakeFS docker image tags by @nicholasjng in #98
• Fix small README mistakes by @nicholasjng in #99

Full Changelog: v0.1.5...v0.1.6

v0.1.5

Fix an Issue that lead to incomplete file uploads for large data files.

What's Changed

• Fix buffer deletes on upload by @nicholasjng in #77
• Only warn once before uploading a raw file by @nicholasjng in #78

Full Changelog: v0.1.4...v0.1.5

v0.1.4

This release introduces fine grained hooks that can be attached to specific FSEvent-events, we improved error messages, and added documentation.

What's Changed

• Better errors by @nicholasjng in #60
• Load and use error message instead of only HTTP reason by @nicholasjng in #61
• Wrap all API calls in try-excepts by @nicholasjng in #62
• Add tests for file operations with prechecks by @nicholasjng in #63
• Add API call wrapper context manager by @nicholasjng in #64
• Add test for expected logs on missing YAML module by @nicholasjng in #65
• Add local S3 block storage adapter docker compose file by @nicholasjng in #69
• Pipe **kwargs to client API calls, make precheck_files a file op keyword argument by @nicholasjng in #70
• Call objects_api.stat_object endpoint in case of file in fs.info by @nicholasjng in #71
• Introduce hooks for fine-grained lakeFS automations in file system operations by @nicholasjng in #72
• Update developer tooling by @nicholasjng in #74
• Fix local S3 docker compose by switching to host networking by @nicholasjng in #75
• Add "reasons to use lakefs-spec" document by @nicholasjng in #73
• Remove link deleted section in README.md by @maxmynter in #76

Full Changelog: v0.1.3...v0.1.4

v0.1.3

This release contains two important bug fixes, namely, adding postcommit support for LakeFSFile writes, and passing the parsed resource to the client in the LakeFSFileSystem.exists() implementation.

What's Changed

• Unpin lakefs-client dependency by @AdrianoKF in #50
• Add implicit branch creation and other small changes to Readme.md by @maxmynter in #48
• Add postcommit support for lakeFS files by @nicholasjng in #51
• Bump multiple tools and dependencies to latest versions by @nicholasjng in #54
• Fix temporary config fixture to back up lakectl config if existent by @nicholasjng in #55
• Add missing tests for fs.exists() and fs.rm() by @nicholasjng in #56

Full Changelog: v0.1.2...v0.1.3

v0.1.2

What's Changed

• Include setuptools-scm for version derivation from Git by @AdrianoKF in #33
• Re-add the note on local lakeFS test instances by @nicholasjng in #34
• Add more configuration arguments to lakeFS file system by @nicholasjng in #37
• Simplify deps by @AdrianoKF in #38
• Improve flexibility of commithooks by adding more arguments by @nicholasjng in #36
• Assert instance caching works by adding a test for it by @nicholasjng in #35
• Update stale commit hook documentation by @nicholasjng in #39
• Remove stale pre-commit configurations, bump ruff to latest v0.0.286 by @nicholasjng in #40
• Record code coverage in CI, upload to Codecov by @AdrianoKF in #41
• Add Tests for implicit Branch Creation by @maxmynter in #42
• chore: Lock main dependencies for CI by @AdrianoKF in #46
• Add lakectl config support by @nicholasjng in #45
• Remove twine check from release workflow by @nicholasjng in #49

Full Changelog: v0.1.0rc5...v0.1.2

v0.1.1

What's Changed

• Include setuptools-scm for version derivation from Git by @AdrianoKF in #33
• Re-add the note on local lakeFS test instances by @nicholasjng in #34
• Add more configuration arguments to lakeFS file system by @nicholasjng in #37
• Simplify deps by @AdrianoKF in #38
• Improve flexibility of commithooks by adding more arguments by @nicholasjng in #36
• Assert instance caching works by adding a test for it by @nicholasjng in #35
• Update stale commit hook documentation by @nicholasjng in #39
• Remove stale pre-commit configurations, bump ruff to latest v0.0.286 by @nicholasjng in #40
• Record code coverage in CI, upload to Codecov by @AdrianoKF in #41
• Add Tests for implicit Branch Creation by @maxmynter in #42
• chore: Lock main dependencies for CI by @AdrianoKF in #46
• Add lakectl config support by @nicholasjng in #45
• Remove twine check from release workflow by @nicholasjng in #49

Full Changelog: v0.1.0rc5...v0.1.1