chore(deps): bump text-splitter from 0.4.4 to 0.6.0

[dependabot]

Bumps text-splitter from 0.4.4 to 0.6.0.

Release notes

Sourced from text-splitter's releases.

v0.6.0

Breaking Changes

• Chunk behavior should now be the same as prior to v0.5.0. Once binary search finds the optimal chunk, we now check the next few sections as long as the chunk size doesn't change. This should result in the same behavior as before, but with the performance improvements of binary search. @​benbrandt in benbrandt/text-splitter#81

Full Changelog: benbrandt/text-splitter@v0.5.1...v0.6.0

v0.5.1

What's New

• Python bindings and Rust crate now have the same version number.

Rust

• Constructors for ChunkSize are now public, so you can more easily create your own ChunkSize structs for your own custom ChunkSizer implementation.

Python

• New CustomTextSplitter that accepts a custom callback with the signature of (str) -> int. Allows for custom chunk sizing on the Python side. @​benbrandt in benbrandt/text-splitter#80

Full Changelog: benbrandt/text-splitter@v0.5.0...v0.5.1

v0.5.0

What's New

• Significant performance improvements for generating chunks with the tokenizers or tiktoken-rs crates by applying binary search when attempting to find the next matching chunk size. @​benbrandt and @​bradfier in benbrandt/text-splitter#71

Breaking Changes

• Minimum required version of tokenizers is now 0.15.0
• Minimum required version of tiktoken-rs is now 0.5.6
• Due to using binary search, there are some slight differences at the edges of chunks where the algorithm was a little greedier before. If two candidates would tokenize to the same amount of tokens that fit within the capacity, it will now choose the shorter text. Due to the nature of of tokenizers, this happens more often with whitespace at the end of a chunk, and rarely effects users who have set with_trim_chunks(true). It is a tradeoff, but would have made the binary search code much more complicated to keep the exact same behavior.
• The chunk_size method on ChunkSizer now needs to accept a ChunkCapacity argument, and return a ChunkSize struct instead of a usize. This was to help support the new binary search method in chunking, and should only affect users who implemented custom ChunkSizers and weren't using one of the provided ones.
  • New signature: fn chunk_size(&self, chunk: &str, capacity: &impl ChunkCapacity) -> ChunkSize;

Full Changelog: benbrandt/text-splitter@v0.4.5...v0.5.0

v0.4.5

What's Changed

• Support tokenizers crate v0.15.0
• Minimum Supported Rust Version is now 1.65.0

New Contributors

• @​FullMetalMeowchemist made their first contribution in benbrandt/text-splitter#53

Full Changelog: benbrandt/text-splitter@v0.4.4...v0.4.5

Changelog

Sourced from text-splitter's changelog.

v0.6.0

Breaking Changes

• Chunk behavior should now be the same as prior to v0.5.0. Once binary search finds the optimal chunk, we now check the next few sections as long as the chunk size doesn't change. This should result in the same behavior as before, but with the performance improvements of binary search.

v0.5.1

What's New

• Python bindings and Rust crate now have the same version number.

Rust

• Constructors for ChunkSize are now public, so you can more easily create your own ChunkSize structs for your own custom ChunkSizer implementation.

Python

• New CustomTextSplitter that accepts a custom callback with the signature of (str) -> int. Allows for custom chunk sizing on the Python side.

v0.5.0

What's New

• Significant performance improvements for generating chunks with the tokenizers or tiktoken-rs crates by applying binary search when attempting to find the next matching chunk size.

Breaking Changes

• Minimum required version of tokenizers is now 0.15.0
• Minimum required version of tiktoken-rs is now 0.5.6
• Due to using binary search, there are some slight differences at the edges of chunks where the algorithm was a little greedier before. If two candidates would tokenize to the same amount of tokens that fit within the capacity, it will now choose the shorter text. Due to the nature of of tokenizers, this happens more often with whitespace at the end of a chunk, and rarely effects users who have set with_trim_chunks(true). It is a tradeoff, but would have made the binary search code much more complicated to keep the exact same behavior.
• The chunk_size method on ChunkSizer now needs to accept a ChunkCapacity argument, and return a ChunkSize struct instead of a usize. This was to help support the new binary search method in chunking, and should only affect users who implemented custom ChunkSizers and weren't using one of the provided ones.
  • New signature: fn chunk_size(&self, chunk: &str, capacity: &impl ChunkCapacity) -> ChunkSize;

v0.4.5

What's New

• Support tokenizers crate v0.15.0
• Minimum Supported Rust Version is now 1.65.0

Commits

• d28f8c0 Fix changelog
• d3acb5e Fix chunk regression with tokenizers (#81)
• 11fe3df Fix failing python docs
• 53cc041 Add CustomTextSplitter for the Python package (#80)
• 320ef56 Bump pyo3 from 0.20.1 to 0.20.2 in /bindings/python (#77)
• eb6d097 Bump pyo3 from 0.20.0 to 0.20.1 in /bindings/python (#76)
• cb35905 Put fits responsibilty in ChunkSize
• f4b049f Fix broken release
• e82517a Python: v0.3.0 release (#73)
• e716aa9 Use binary search in TextChunks (#71)
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

[dependabot]

Superseded by #96.