Update text-splitter requirement from 0.24.1 to 0.25.1

[dependabot]

Updates the requirements on text-splitter to permit the latest version.

Release notes

Sourced from text-splitter's releases.

v0.25.1

What's New

• Use memchr crate instead of regex for parsing phase in TextSplitter. This should improve performance in how quickly the text is parsed when scanning for newline characters. benbrandt/text-splitter#650
• Implement ChunkSizer trait automatically for many more wrappers and references to types that already implement ChunkSizer benbrandt/text-splitter#649

Full Changelog: benbrandt/text-splitter@v0.25.0...v0.25.1

Changelog

Sourced from text-splitter's changelog.

v0.25.1

What's New

• Use memchr crate instead of regex for parsing phase in TextSplitter. This should improve performance in how quickly the text is parsed when scanning for newline characters.
• Implement ChunkSizer trait automatically for many more wrappers and references to types that already implement ChunkSizer.

v0.25.0

Breaking Changes

Rust

• Remove support for rust-tokenizers crate. This crate hasn't been updated in several years and brings in depednencies that have security warnings.

What's New

• Use faster encoding method for tokenizers library, which improves performance with usage of huggingface tokenizers.

v0.24.2

Fixes

• Python packages target a newer version of libc to hopefully fix header file issues with tree-sitter.

What's New

• MSRV updated to 1.81.0

v0.24.1

What's New

Added a new chunk_char_indices method to the Rust splitters.

use text_splitter::{Characters, ChunkCharIndex, TextSplitter};
let text = "\r\na̐éö̲\r\n";
let splitter = TextSplitter::new(3);
let chunks = splitter.chunk_char_indices(text).collect::<Vec<_>>();
assert_eq!(
vec![
ChunkCharIndex {
chunk: "a̐é",
byte_offset: 2,
char_offset: 2
},
ChunkCharIndex {
</tr></table>

... (truncated)

Commits

• 0e8c166 chore: attempt to lower requirement on memchr
• c2c5fa4 docs: fix comment style
• a0d1da1 feat: Replace regex with memchr for linebreak detection
• 37b6a83 ci: add cache to coverage
• ca5d75d test: Add test for chunk size reference impl
• 7b77da1 docs: Fix doc comment style in text splitter implementation
• c82a5fd test: test chunk sizer usage instead of just dropping config
• 48e5e56 chore: prep release
• b36126d ci: fix python caching
• 4ca54c3 feat: support many more auto trait impls for ChunkSizers
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)