v2.0.0-beta.1 – Orchestrating Knowledge, Powering Workflows

[laipz8200]

🚀 Introduction

In Dify 2.0, we are introducing two major new capabilities: the Knowledge Pipeline and the Queue-based Graph Engine.

This is a beta release, and we hope to explore these improvements together with you and gather your feedback. The Knowledge Pipeline provides a modularized and extensible workflow for knowledge ingestion and processing, while the Queue-based Graph Engine makes workflow execution more robust and controllable. We believe these will help you build and debug AI applications more smoothly, and we look forward to your experiences to help us continuously improve.

---

📚 Knowledge Pipeline

✨ Introduction

With the brand-new orchestration interface for knowledge pipelines, we introduce a fundamental architectural upgrade that reshapes how document processing are designed and executed, providing a more modular and flexible workflow that enables users to orchestrate every stage of the pipeline. Enhanced with a wide range of powerful plugins available in the marketplace, it empowers users to flexibly integrate diverse data sources and processing tools. Ultimately, this architecture enables building highly customized, domain-specific RAG solutions that meet enterprises’ growing demands for scalability, adaptability, and precision.

❓ Why Do We Need It?

Previously, Dify's RAG users still encounter persistent challenges in real-world adoption — from inaccurate knowledge retrieval and information loss to limited data integration and extensibility. Common pain points include:

• 🔗 restricted integration of data sources
• 🖼️ missing critical elements such as tables and images
• ✂️ suboptimal chunking results

All of them lead to poor answer quality and hinder the model's overall performance.

In response, we reimagined RAG in Dify as an open and modular architecture, enabling developers, integrators, and domain experts to build document processing pipelines tailored to their specific requirements—from data ingestion to chunk storage and retrieval.

🛠️ Core Capabilities

🧩 Knowledge Pipeline Architecture

The Knowledge Pipeline is a visual, node-based orchestration system dedicated to document ingestion. It provides a customizable way to automate complex document processing, enabling fine-grained transformations and bridging raw content with structured, retrievable knowledge. Developers can build workflows step by step, like assembling puzzle pieces, making document handling easier to observe and adjust.

📑 Templates & Pipeline DSL

• ⚡ Start quickly with official templates
• 🔄 Customize and share pipelines by importing/exporting via DSL for easier reusability and collaboration

🔌 Customizable Data Sources & Tools

Each knowledge base can support multiple data sources. You can seamlessly integrate local files, online documents, cloud drives, and web crawlers through a plugin-based ingestion framework. Developers can extend the ecosystem with new data-source plugins, while marketplace processors handle specialized use cases like formulas, spreadsheets, and image parsing — ensuring accurate ingestion and structured representation.

🧾 New Chunking Strategies

In addition to General and Parent-Child modes, the new Q&A Processor plugin supports Q&A structures. This expands coverage for more use cases, balancing retrieval precision with contextual completeness.

🖼️ Image Extraction & Retrieval

Extract images from documents in multiple formats, store them as URLs in the knowledge base, and enable mixed text-image outputs to improve LLM-generated answers.

🧪 Test Run & Debugging Support

Before publishing a pipeline, you can:

• ▶️ Execute a single step or node independently
• 🔍 Inspect intermediate variables in detail
• 👀 Preview string variables as Markdown in the variable inspector

This provides safe iteration and debugging at every stage.

🔄 One-Click Migration from Legacy Knowledge Bases

Seamlessly convert existing knowledge bases into the Knowledge Pipeline architecture with a single action, ensuring smooth transition and backward compatibility.

🌟 Why It Matters

The Knowledge Pipeline makes knowledge management more transparent, debuggable, and extensible. It is not the endpoint, but a foundation for future enhancements such as multimodal retrieval, human-in-the-loop collaboration, and enterprise-level data governance. We’re excited to see how you apply it and share your feedback.

---

⚙️ Queue-based Graph Engine

❓ Why Do We Need It?

Previously, designing workflows with parallel branches often led to:

• 🌀 Difficulty managing branch states and reproducing errors
• ❌ Insufficient debugging information
• 🧱 Rigid execution logic lacking flexibility

These issues reduced the usability of complex workflows. To solve this, we redesigned the execution engine around queue scheduling, improving management of parallel tasks.

🛠️ Core Capabilities

📋 Queue Scheduling Model

All tasks enter a unified queue, where the scheduler manages dependencies and order. This reduces errors in parallel execution and makes topology more intuitive.

🎯 Flexible Execution Start Points

Execution can begin at any node, supporting partial runs, resumptions, and subgraph invocations.

🌊 Stream Processing Component

A new ResponseCoordinator handles streaming outputs from multiple nodes, such as token-by-token LLM generation or staged results from long-running tasks.

🕹️ Command Mechanism

With the CommandProcessor, workflows can be paused, resumed, or terminated during execution, enabling external control.

🧩 GraphEngineLayer

A new plugin layer that allows extending engine functionality without modifying core code. It can monitor states, send commands, and support custom monitoring.

---

Quickstart

1. Prerequisites
   • Dify version: 2.0.0-beta.1 or higher
2. How to Enable
   • Enabled by default, no additional configuration required.
   • Debug mode: set DEBUG=true to enable DebugLoggingLayer.
   • Execution limits:
     • WORKFLOW_MAX_EXECUTION_STEPS=500
     • WORKFLOW_MAX_EXECUTION_TIME=1200
     • WORKFLOW_CALL_MAX_DEPTH=10
   • Worker configuration (optional):
     • WORKFLOW_MIN_WORKERS=1
     • WORKFLOW_MAX_WORKERS=10
     • WORKFLOW_SCALE_UP_THRESHOLD=3
     • WORKFLOW_SCALE_DOWN_IDLE_TIME=30
   • Applies to all workflows.

---

More Controllable Parallel Branches

Execution Flow:

Start ─→ Unified Task Queue ─→ WorkerPool Scheduling
                          ├─→ Branch-1 Execution
                          └─→ Branch-2 Execution
                                  ↓
                            Aggregator
                                  ↓
                                  End

Improvements:
1. All tasks enter a single queue, managed by the Dispatcher.
2. WorkerPool auto-scales based on load.
3. ResponseCoordinator manages streaming outputs, ensuring correct order.

---

Example: Command Mechanism

from core.workflow.graph_engine.manager import GraphEngineManager

# Send stop command
GraphEngineManager.send_stop_command(
    task_id="workflow_task_123",
    reason="Emergency stop: resource limit exceeded"
)

Note: pause/resume functionality will be supported in future versions.

---

Example: GraphEngineLayer

GraphEngineLayer Example

---

FAQ

1. Is this release focused on performance?
   No. The focus is on stability, clarity, and correctness of parallel branches. Performance improvements are a secondary benefit.

2. What events can be subscribed to?

   • Graph-level: GraphRunStartedEvent, GraphRunSucceededEvent, GraphRunFailedEvent, GraphRunAbortedEvent
   • Node-level: NodeRunStartedEvent, NodeRunSucceededEvent, NodeRunFailedEvent, NodeRunRetryEvent
   • Container nodes: IterationRunStartedEvent, IterationRunNextEvent, IterationRunSucceededEvent, LoopRunStartedEvent, LoopRunNextEvent, LoopRunSucceededEvent
   • Streaming output: NodeRunStreamChunkEvent

3. How can I debug workflow execution?

   • Enable DEBUG=true to view detailed logs.
   • Use DebugLoggingLayer to record events.
   • Add custom monitoring via GraphEngineLayer.

---

Future Plans

This beta release is just the beginning. Upcoming improvements include:

• Debugging Tools: A visual interface to view execution states and variables in real time.
• Intelligent Scheduling: Optimize scheduling strategies using historical data.
• More Complete Command Support: Add Pause/Resume, breakpoint debugging.
• Human in the Loop: Support human intervention during execution.
• Subgraph Functionality: Enhance modularity and reusability.
• Multimodal Embedding: Support richer content types beyond text.

We look forward to your feedback and experiences to make the engine more practical.

---

Upgrade Guide

Important

After upgrading, you must run the following migration to transform existing datasource credentials. This step is required to ensure compatibility with the new version:

uv run flask transform-datasource-credentials

Docker Compose Deployments

1. Back up your customized docker-compose YAML file (optional)

cd docker
cp docker-compose.yaml docker-compose.yaml.$(date +%s).bak

2. Get the latest code from the main branch

git checkout 2.0.0-beta.1
git pull origin 2.0.0-beta.1

3. Stop the service. Please execute in the docker directory

docker compose down

4. Back up data

tar -cvf volumes-$(date +%s).tgz volumes

5. Upgrade services

docker compose up -d

6. Migrate data after the container starts

docker exec -it docker-api-1 uv run flask transform-datasource-credentials

Source Code Deployments

1. Stop the API server, Worker, and Web frontend Server.

2. Get the latest code from the release branch:

git checkout 2.0.0-beta.1

3. Update Python dependencies:

cd api
uv sync

4. Then, let's run the migration script:

uv run flask db upgrade
uv run flask transform-datasource-credentials

5. Finally, run the API server, Worker, and Web frontend Server again.

---

This discussion was created from the release v2.0.0-beta.1 – Orchestrating Knowledge, Powering Workflows.

[DavideDelbianco]

Currently I do something similar with 2 workflows transformed as tools.

I call them "KB Upsert" and "KB Embed Doc"
The first one accepts a dify API URL and Key, the KB name, the metadata definitions.
When called:

• Uses API calls to search the KB
• Creates it if it does not exist
• Enables default metadata
• if metadata definitions were provided, cycles them and creates them
• Returns the dataset ID

KB Embed Doc:
Receives API URL and Key, Dataset ID, document payload (embed by text API payload), metadata definitions
When called:

• Execute the embedding API call
• adds metadata do the document

By now it should be clear what my point is...
The node that executes the embedding, MUST allow metadata binding.
Metadata is important to increase retrieval accuracy and quality. Successfully filtering docs also improves retrieval speed.

What about retrieval?

This new pipeline simplifies document embedding, but keeps a wrong concept of the previous RAG:
Retrieval strategy is bound to the KB and cannot be altered at retrieval time.

It would be useful in the KB retrieval node to be able to change retrieval configuration by choosing at least the top K value without altering it for any other retrieval node that uses the same KB.
Depending on the vector DB, changing top K greatly alters the results.
(Being able to opt to use rerank or not, hydrib or not, would be a plus)

As I've suggested in ehancement issues, we get parameters and customization for LLM models used in text generation, but there are parameters for Embedding and Reranking models too, to alter how the model performs, I've already mentioned in the past the Matryoshka settings

Another improvement:

It should be possible to trigger the pipeline by API and by trigger nodes (when implemented) to allow a pipeline to use the HTTP node / any other node, including workflows as tool (deep search workflow) to gather data to embed.

More RAG methods

Will you implement Graph RAG ?

[Emunhuu88]

Enhancing retrieval would be great improvement

[DavideDelbianco]

In the future improvements you included:

• sub workflow (great, but make tracing, and logs consistent, because when invoking sub graphs by workflow as tool it becomes harder to debug)

What is needed in sub workflows and workflow as tool?
A node to instantly return and keep workflow going.
This way the main workflow will consider the node as executed, while the workflow keeps doing something in background.
The most noticeable use is executing something after the answer node or in parallel to the end node, for example to send a report by mail, or log something in an external platform.
The workflow is considered still running and you cannot send new messages, I should be able to decide if I want to consider / signal the execution as completed.

[Emunhuu88]

With knowledge pipeline being added to DIFY, you guys should consider improving the retrieval part as well. Being limited to max of 10 retrieved chunk is limiting. I am sure there are a lot of retrieval methods in space of RAG.

[reymondpamelar]

your max retrieved chunks can be adjusted within the environment variables. I believe its:

TOP_K_MAX_VALUE: '100'

[cyberbobjr]

How about insert/upsert metadata ? with the v1.8.x i do it with my custom python code and using Dify KB API, but do you plan to integrate metadata management workflow in the future ? (metadata from my part is almost as important as chunking for retrieval part)

[arsenetripard]

Same concern, Strongly agree :)

[aldywxr]

chatflow无法访问sys.query。报错：Run failed: cannot access local variable 'curr_message_tokens' where it is not associated with a value。

[laipz8200]

Fixed in beta.2

[reymondpamelar]

Parallel iteration does not seem to be working anymore. Enabling parallelism will execute one iteration at a time.

Note: My env variables are unchanged since v1.8.1

[laipz8200]

I've added this function to the new engine, it will be released in next version.

[aldywxr]

I've tried it, but it still doesn't work. RayzZZ ***@***.*** 原始邮件 发件人：Reymond Pamelar ***@***.***> 发件时间：2025年9月9日 00:58 收件人：langgenius/dify ***@***.***> 抄送：Ray ***@***.***>, Comment ***@***.***> 主题：Re: [langgenius/dify] v2.0.0-beta.1 – Orchestrating Knowledge, Powering Workflows (Discussion #25176) your max retrieved chunks can be adjusted within the environment variables. I believe its: TOP_K_MAX_VALUE: '100' — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

[ccsang]

Congratulations on shipping Dify 2.0! The release looks fantastic and the improvements are much appreciated. Could you please share when the Human-in-the-Loop capability is expected to be available (beta or GA)? Even a rough window would help us plan our rollout. Many thanks for all the great work.