How to disable keywords?

[secsilm]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

I am using the create_by_text API to upload my documents (around 22,000). The documents have been uploaded, but the vectors are still queued for computation (very slow, seems to be calculated one by one rather than in parallel).

Last time, I uploaded about 40,000 documents, which resulted in filling up my hard drive space (using over 100GB, while my original files were only around 200MB).

This time, I conducted further observations and found that the dataset_keyword_tables table occupies a very high amount of space, which can be said to be largely occupied by this table.

I suspect that this table is used to store the keywords corresponding to each document, but I only use vector search. How can I disable keywords? Or is there any way to reduce this table's hard drive usage?

2. Additional context or comments

Related issue: #6269

[secsilm]

Again, dify eated all my storage.

[taverngoal]

me too, keywords seems not necessary in somet scene

[imaben]

The same issue.

Request administrator attention to a serious issue.

[shawn-maxiao]

same issue.
I have read the source to find why.
I find the extract keywords function very slow( 2~5 seconds ), the function uses Jieba to extract keywords for every document chunk.
And there is one redis lock, which makes the other extract-keywords threads wait for the extract-keywords thread which is executing, this causes the process looks like one-by-one instead of parallel.
How to solve it?

[glacierck]

You have identified the issue, but it is not precise enough. In fact, the data for 11G is all recorded on one line. That's right, one grid stores the keyword data object JSON of your entire knowledge base. The system needs to repeatedly serialize and deserialize this 11g JSON.
Here is my rescue method: define a function in the database to listen for and delete writes to this table.

CREATE OR REPLACE FUNCTION "public"."delete_on_insert"()
  RETURNS "pg_catalog"."trigger" AS $BODY$
BEGIN
    -- 删除新插入的记录
    DELETE FROM dataset_keyword_tables WHERE id = NEW.id;
    -- 返回NULL，表示触发器已成功处理事件，不需要进一步处理
    RETURN NULL;
END;
$BODY$
  LANGUAGE plpgsql VOLATILE
  COST 100

[glacierck]

I can't imagine why this is implemented in a way that only solves the problem but is destined to be unusable if it is important data. It looks like intentional destruction and planting landmines. It's a mistake that normal beginners wouldn't make, but when it appears, we have to suspect it was intentional. If the official hosting doesn't have this problem, then this incident would be funny.

[glacierck]

There is another doubt, this issue has been exposed for over a year now！

[xdmushui]

Why this issue still exists?