✨ feat: Add configurable PDF processing method with Unstructured #5927

fzlzjerry · 2025-02-09T05:22:33Z

💻 变更类型 | Change Type

🔀 变更说明 | Description of Change

📝 补充信息 | Additional Information

vercel · 2025-02-09T05:22:37Z

@fzlzjerry is attempting to deploy a commit to the LobeHub Team on Vercel.

A member of the Team first needs to authorize it.

lobehubbot · 2025-02-09T05:22:47Z

👍 @fzlzjerry

Thank you for raising your pull request and contributing to our Community
Please make sure you have followed our contributing guidelines. We will review it as soon as possible.
If you encounter any problems, please feel free to connect with us.
非常感谢您提出拉取请求并为我们的社区做出贡献，请确保您已经遵循了我们的贡献指南，我们会尽快审查它。
如果您遇到任何问题，请随时与我们联系。

gru-agent · 2025-02-09T05:22:50Z

TestGru Assignment

Summary

Link	CommitId	Status	Reason
Detail	`06cbc84`	✅ Finished

Files

File	Pull Request
src/server/modules/ContentChunk/index.ts	❌ Failure (I failed to write the unit tests for the file.)
src/server/utils/env.ts	🔴 Closed #5930

Tip

You can @gru-agent and leave your feedback. TestGru will make adjustments based on your input

src/server/modules/ContentChunk/index.ts

arvinxx · 2025-02-13T03:01:23Z

src/server/modules/ContentChunk/index.ts

@@ -26,7 +31,10 @@ export class ContentChunk {
  }

  isUsingUnstructured(params: ChunkContentParams) {
-    return params.fileType === 'application/pdf' && params.mode === 'hi-res';
+    return params.fileType === 'application/pdf' &&


另外这里有个点想探讨一下，你觉得是只有 PDF 都走 Unstructured ，还是说我们可以给出一个扩展语法，比如:

CHUNK_BY_UNSTRUCTURED=pdf,xlsx

这样，使得我们可以灵活控制哪些文件走 Unstructured 还是用原生解析。

还是现在这个 USE_UNSTRUCTURED_FOR_PDF，只考虑 pdf 的使用场景？

灵活控制？毕竟有些docs这些也是有文件的

那要不这里改成按规则控制的实现吧？

这个变量名可以改成 FILE_TYPE_CHUNKING_RULES

然后按照 key=value 的形式来写？

比如 PDF 和 XLSX 走 UnstructuredIO，那么变量就是：

FILE_TYPE_CHUNKING_RULES="pdf=unstructured;xlsx=unstructured"

如果未来还有别的 PDF Chunking 服务商，那么就可以灵活控制：

FILE_TYPE_CHUNKING_RULES="pdf=unstructured,doc2x,default;xlsx=unstructured"

这就意味着 PDF 解析的服务有 3 种： unstructured 、 doc2x 和默认，然后如果 unstructured 解析失败则往下调用 doc2x 来解析，如果 doc2x 还失败的话，则兜底到内建 default 解析。而 xlsx 没写 default 的话，就只用 unstructured 解析

…config

src/server/utils/env.ts

arvinxx · 2025-02-13T10:17:13Z

src/server/modules/ContentChunk/index.ts

@@ -26,7 +31,10 @@ export class ContentChunk {
  }

  isUsingUnstructured(params: ChunkContentParams) {
-    return params.fileType === 'application/pdf' && params.mode === 'hi-res';
+    return params.fileType === 'application/pdf' &&


那要不这里改成按规则控制的实现吧？

这个变量名可以改成 FILE_TYPE_CHUNKING_RULES

然后按照 key=value 的形式来写？

比如 PDF 和 XLSX 走 UnstructuredIO，那么变量就是：

FILE_TYPE_CHUNKING_RULES="pdf=unstructured;xlsx=unstructured"

如果未来还有别的 PDF Chunking 服务商，那么就可以灵活控制：

FILE_TYPE_CHUNKING_RULES="pdf=unstructured,doc2x,default;xlsx=unstructured"

这就意味着 PDF 解析的服务有 3 种： unstructured 、 doc2x 和默认，然后如果 unstructured 解析失败则往下调用 doc2x 来解析，如果 doc2x 还失败的话，则兜底到内建 default 解析。而 xlsx 没写 default 的话，就只用 unstructured 解析

arvinxx · 2025-02-13T16:41:33Z

src/server/modules/ContentChunk/index.ts

    return await this.chunkByLangChain(params.filename, params.content);
  }

+  private canUseUnstructured(): boolean {
+    return !!(
+      knowledgeEnv.USE_UNSTRUCTURED_FOR_PDF &&


是不是不需要 USE_UNSTRUCTURED_FOR_PDF 这个变量了？

arvinxx · 2025-02-13T16:41:56Z

src/config/knowledge.ts

+    UNSTRUCTURED_API_KEY: process.env.UNSTRUCTURED_API_KEY,
+    UNSTRUCTURED_SERVER_URL: process.env.UNSTRUCTURED_SERVER_URL,
+    USE_UNSTRUCTURED_FOR_PDF: process.env.USE_UNSTRUCTURED_FOR_PDF,
+    FILE_TYPE_CHUNKING_RULES: process.env.FILE_TYPE_CHUNKING_RULES || 'pdf=unstructured',


这里不应该加 'pdf=unstructured' 默认值吧？

arvinxx · 2025-02-13T16:43:19Z

src/server/modules/ContentChunk/rules.ts

让 ai 生成一波单测？保证下稳定性。参考： https://github.com/lobehub/lobe-chat/blob/main/src/server/globalConfig/parseFilesConfig.test.ts

…configuration

fzlzjerry added 2 commits February 9, 2025 12:42

✨ feat: Add configurable PDF processing method with Unstructured

8745ca5

🔧 fix: Update import path for env utility in ContentChunk module

06cbc84

dosubot bot added size:S This PR changes 10-29 lines, ignoring generated files. 🌠 Feature Request New feature or request | 特性与建议 labels Feb 9, 2025

Merge branch 'main' into fix/unstructured_io

76bdec6

gru-agent bot mentioned this pull request Feb 9, 2025

test: add unit test for src/server/utils/env.ts #5930

Closed

fzlzjerry added 3 commits February 9, 2025 14:34

Merge branch 'main' into fix/unstructured_io

d868e7b

Merge branch 'main' into fix/unstructured_io

f9b7751

Merge branch 'main' into fix/unstructured_io

c67fb33

arvinxx requested changes Feb 13, 2025

View reviewed changes

fzlzjerry added 2 commits February 13, 2025 13:04

Merge branch 'lobehub:main' into fix/unstructured_io

e8f0b03

feat: add USE_UNSTRUCTURED_FOR_PDF environment variable to knowledge …

4de5511

…config

dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:S This PR changes 10-29 lines, ignoring generated files. labels Feb 13, 2025

fzlzjerry requested a review from arvinxx February 13, 2025 07:37

Merge branch 'main' into fix/unstructured_io

79a6da5

arvinxx requested changes Feb 13, 2025

View reviewed changes

fzlzjerry added 2 commits February 13, 2025 19:02

Delete src/server/utils/env.ts

d6301bb

feat: implement ChunkingRuleParser for file type and service mapping

c132ff3

dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:M This PR changes 30-99 lines, ignoring generated files. labels Feb 13, 2025

arvinxx reviewed Feb 13, 2025

View reviewed changes

fzlzjerry added 4 commits February 14, 2025 20:14

Merge branch 'main' into fix/unstructured_io

6c3f3f4

refactor: remove USE_UNSTRUCTURED_FOR_PDF from knowledge environment …

b82f9fe

…configuration

test: add unit tests for ChunkingRuleParser functionality

32771f5

Merge branch 'main' into fix/unstructured_io

ef63c8a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ feat: Add configurable PDF processing method with Unstructured #5927

✨ feat: Add configurable PDF processing method with Unstructured #5927

fzlzjerry commented Feb 9, 2025 •

edited

Loading

vercel bot commented Feb 9, 2025

lobehubbot commented Feb 9, 2025

gru-agent bot commented Feb 9, 2025 •

edited

Loading

arvinxx Feb 13, 2025

fzlzjerry Feb 13, 2025

arvinxx Feb 13, 2025 •

edited

Loading

arvinxx Feb 13, 2025 •

edited

Loading

arvinxx Feb 13, 2025 •

edited

Loading

arvinxx Feb 13, 2025

arvinxx Feb 13, 2025

✨ feat: Add configurable PDF processing method with Unstructured #5927

Are you sure you want to change the base?

✨ feat: Add configurable PDF processing method with Unstructured #5927

Conversation

fzlzjerry commented Feb 9, 2025 • edited Loading

💻 变更类型 | Change Type

🔀 变更说明 | Description of Change

📝 补充信息 | Additional Information

vercel bot commented Feb 9, 2025

lobehubbot commented Feb 9, 2025

gru-agent bot commented Feb 9, 2025 • edited Loading

TestGru Assignment

Summary

Files

arvinxx Feb 13, 2025

Choose a reason for hiding this comment

fzlzjerry Feb 13, 2025

Choose a reason for hiding this comment

arvinxx Feb 13, 2025 • edited Loading

Choose a reason for hiding this comment

arvinxx Feb 13, 2025 • edited Loading

Choose a reason for hiding this comment

arvinxx Feb 13, 2025 • edited Loading

Choose a reason for hiding this comment

arvinxx Feb 13, 2025

Choose a reason for hiding this comment

arvinxx Feb 13, 2025

Choose a reason for hiding this comment

fzlzjerry commented Feb 9, 2025 •

edited

Loading

gru-agent bot commented Feb 9, 2025 •

edited

Loading

arvinxx Feb 13, 2025 •

edited

Loading

arvinxx Feb 13, 2025 •

edited

Loading

arvinxx Feb 13, 2025 •

edited

Loading