Encountered issue using OpenAI Tokenizer Wrapper

[Abdullahkhan5]

While executing the code with OpenAI Tokenizer Wrapper I encountered the following issue:

Traceback (most recent call last):
File "E:\AI Governance Sandbox\RAG - Docling\ai-cookbook\knowledge\docling\2-chunking.py", line 34, in
chunker = HybridChunker(
^^^^^^^^^^^^^^
File "E:\AI Governance Sandbox\RAG - Docling\ai-cookbook\knowledge\docling\venv\Lib\site-packages\pydantic\main.py", line 253, in init
validated_self = self.pydantic_validator.validate_python(data, self_instance=self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI Governance Sandbox\RAG - Docling\ai-cookbook\knowledge\docling\venv\Lib\site-packages\docling_core\transforms\chunker\hybrid_chunker.py", line 73, in _patch
if isinstance(data, dict) and (tokenizer := data.get("tokenizer")):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\AI Governance Sandbox\RAG - Docling\ai-cookbook\knowledge\docling\venv\Lib\site-packages\transformers\tokenization_utils_base.py", line 1513, in len
raise NotImplementedError()
NotImplementedError

[venkateshgbabu]

I get the exact same error.

`File /opt/anaconda3/envs/docling-ex/lib/python3.13/site-packages/docling_core/transforms/chunker/hybrid_chunker.py:73, in HybridChunker._patch(cls, data)
70 @model_validator(mode="before")
71 @classmethod
72 def _patch(cls, data: Any) -> Any:
---> 73 if isinstance(data, dict) and (tokenizer := data.get("tokenizer")):
74 max_tokens = data.get("max_tokens")

File /opt/anaconda3/envs/docling-ex/lib/python3.13/site-packages/transformers/tokenization_utils_base.py:1513, in PreTrainedTokenizerBase.len(self)
1512 def len(self) -> int:
-> 1513 raise NotImplementedError()

NotImplementedError: `

[Abdullahkhan5]

@venkateshgbabu you can fix it by modifying the utils file like this, hope this helps!
https://github.com/daveebbelaar/ai-cookbook/pull/7/files