Feature/improve docs #1051

emrgnt-cmplxty · 2024-09-05T00:30:21Z

🚀	This description was created by Ellipsis for commit `7d9564e`

Summary:

This PR enhances R2R documentation and refines the ingestion process with improved file handling and parsing logic.

Key points:

Improved R2R documentation with updated hybrid search details and enhanced introduction.
Restructured navigation for better clarity and usability.
Added Docker and local system installation instructions.
Updated ingestion process with improved file handling and metadata management.
Enhanced parsing logic and logging in parsing providers.
Improved error handling and metadata updates in ingestion services.

Generated with ❤️ by ellipsis.dev

ellipsis-dev

❌ Changes requested. Reviewed everything up to 5e1c9e6 in 24 seconds

More details

Looked at 696 lines of code in 14 files
Skipped 0 files when reviewing.
Skipped posting 0 drafted comments based on config settings.

Workflow ID: wflow_5mLnWtdBAFYXEUAS

Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

ellipsis-dev · 2024-09-05T00:31:17Z

py/core/base/providers/__init__.py

@@ -29,6 +29,7 @@
    "AuthConfig",
    "AuthProvider",
    # Chunking provider
+    "UnstructuredChunkingConfig",
    "ChunkingConfig",
    "ChunkingProvider",
    "UnstructuredChunkingConfig",


UnstructuredChunkingConfig is listed twice in the __all__ list. Remove the duplicate entry.

ellipsis-dev · 2024-09-05T00:31:17Z

py/core/__init__.py

@@ -138,6 +138,7 @@
    "AuthConfig",
    "AuthProvider",
    # Chunking provider
+    "UnstructuredChunkingConfig",
    "ChunkingConfig",
    "ChunkingProvider",
    "Method",


UnstructuredChunkingConfig is listed twice in the __all__ list. Remove the duplicate entry.

vercel · 2024-09-05T02:31:57Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
recommendation_platform	⬜️ Ignored (Inspect)			Sep 5, 2024 2:31am

ellipsis-dev

👍 Looks good to me! Incremental review on 7d9564e in 22 seconds

More details

Looked at 544 lines of code in 10 files
Skipped 0 files when reviewing.
Skipped posting 3 drafted comments based on config settings.

1. py/core/providers/parsing/r2r_parsing.py:84

Draft comment:
Remove print statements or replace them with logging for better production practices.
Reason this comment was not posted:
Confidence changes required: 50%
The print statements in the code are likely for debugging purposes and should be removed or replaced with proper logging.

2. py/core/providers/parsing/unstructured_parsing.py:116

Draft comment:
Remove print statements or replace them with logging for better production practices.
Reason this comment was not posted:
Confidence changes required: 50%
The print statements in the code are likely for debugging purposes and should be removed or replaced with proper logging.

3. py/core/providers/embeddings/litellm.py:67

Draft comment:
Remove print statements or replace them with logging for better production practices.
Reason this comment was not posted:
Confidence changes required: 50%
The print statements in the code are likely for debugging purposes and should be removed or replaced with proper logging.

Workflow ID: wflow_veQsYFCfIdc4qdpq

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

* Feature/orchestration v0 (#1006) * Feature/remove extra r2r abstraction (#996) * moving kg construction to enrich-graph (#984) * checkin * up * done * formatting * Feature/update ingestion issues (#985) * udpate ingestion issues * keep unbounded limit support, but default to bounded * fix * fmt * removes an unnecessary abstraction * sync changes --------- Co-authored-by: Shreyas Pimpalgaonkar <[email protected]> * first commit * move towards orchestration * tweaks * check in working ingestion * move * kg enrichment * update future, postgres compose * hatchetize ingestion pipeline * ready for prime time * finish --------- Co-authored-by: Shreyas Pimpalgaonkar <[email protected]> * Feature/add update files workflow (#1010) * add update files workflow * rm ingestion pipeline * Feature/add enrichment flow (#1013) * add update files workflow * rm ingestion pipeline * v0 restructure orch * Feature/merged enrichment flow (#1016) * add update files workflow * rm ingestion pipeline * v0 restructure orch * kg orchestration * finish kg orchestration * update service * merge * cleanups * Rm graspologic (#1034) * moving kg construction to enrich-graph (#984) * checkin * up * done * formatting * Feature/update ingestion issues (#985) * udpate ingestion issues * keep unbounded limit support, but default to bounded * fix * fmt * Add support for CharacterTextSplitter (#986) * Add support for CharacterTextSplitter Allows R2R client to override the text splitter. Example: ```python ingestion_response = client.ingest_files( file_paths=[file_path], metadatas=metadata, # optionally override chunking settings at runtime chunking_settings={ "provider": "r2r", "method": "character", "extra_fields": { "separator": "---" }, } ) ``` * fixup! Add support for CharacterTextSplitter * fixup! fixup! Add support for CharacterTextSplitter * Patch/ollama base cli (#992) * Dev (#990) * moving kg construction to enrich-graph (#984) * checkin * up * done * formatting * Feature/update ingestion issues (#985) * udpate ingestion issues * keep unbounded limit support, but default to bounded * fix * fmt * Add support for CharacterTextSplitter (#986) * Add support for CharacterTextSplitter Allows R2R client to override the text splitter. Example: ```python ingestion_response = client.ingest_files( file_paths=[file_path], metadatas=metadata, # optionally override chunking settings at runtime chunking_settings={ "provider": "r2r", "method": "character", "extra_fields": { "separator": "---" }, } ) ``` * fixup! Add support for CharacterTextSplitter * fixup! fixup! Add support for CharacterTextSplitter --------- Co-authored-by: Shreyas Pimpalgaonkar <[email protected]> Co-authored-by: Manuel R. Ciosici <[email protected]> * fix ollama cli --------- Co-authored-by: Shreyas Pimpalgaonkar <[email protected]> Co-authored-by: Manuel R. Ciosici <[email protected]> * Ingestion refactor (#991) * fix test (#993) * Increase Neo4j memory limits, add GDS plugin, and update LLM concurrency limit to 256. * Update ingestion sample file, disable KG node extraction pipe, add community processing in clustering, and enhance graph clustering queries. * Update runners (#1007) * Refactor KG clustering process to simplify community processing and enhance entity-triple retrieval from Neo4j. * Refactor Neo4j configuration for memory settings and update graph clustering logic in the KG provider. * Fix pipeline by enabling node extraction and refactor community processing logic in KGClusteringPipe. * hatchet works * throw error if you run global search before enrichment * Fix communities in local search * turn off node desc embedding * fix rag endpoint * Increase hatchet msg size * Update ingestion.py * Refactor and clean up code formatting * modified workflow * Add graph creation functionality * Refactor KG parameters and logging. * review * up --------- Co-authored-by: emrgnt-cmplxty <[email protected]> Co-authored-by: emrgnt-cmplxty <[email protected]> Co-authored-by: Manuel R. Ciosici <[email protected]> Co-authored-by: Nolan Tremelling <[email protected]> * Feature/add hatchet api key setup rebased (#1040) * add update files workflow * rm ingestion pipeline * v0 restructure orch * kg orchestration * finish kg orchestration * update service * merge * cleanups * add hatchet api key setup * cleanup * add hatchet api key setup (#1037) * add hatchet api key setup * cleanup * fix merge * cleanups * Feature/nolan logs refactored (#1041) * Update runners (#1007) * Check in logs --------- Co-authored-by: Nolan Tremelling <[email protected]> * Pull open PRs into dev (#1042) * Pull in subnet and graph PR * Add in templates * Add python files for templates in cli (#1043) * working hatchet integration (#1046) * Update local_llm_neo4j_kg.toml * Unstructured fixes (#1048) * dockerfile * Update ingestion file with new sample URL and enhance unstructured chunking configuration and error handling. * clean up * clean up dockerfile * up * Update sample file and clean code * Add hatchet-sdk dependency in project. * Update providers to include local option. * Introduce File Provider (#1044) * Draft of file provider * Some cleanup * Regenearte lock * Stream it * Use document_id as primary key * Pydantic v2 * File provider finished * Make 7272 the default port (#1045) * Fix poetry.lock * Precommit * Enhance Dockerfile and add telemetry events (#1049) * Fix File Provider (#1050) * Fix * Fix parsing pipeline * working * Feature/improve docs (#1051) * improve documentation * fix unstr * add ingestion * fix compose * Add unstructured chunking configuration updates * Revert "Add unstructured chunking configuration updates" This reverts commit bae8c0b. * Separate File Provider and Relational Database Provider (#1054) * Move to self.execute_query * Check in push * Check in * Get file provider running * Actually use file provider * Final touches * undo changes in compose * Patch/fix unstructured config rebased (#1059) * fix unstr err * tweak * by_title default * cleanups * checkin * merge * Graph docs (#1058) * Add document chunks and enrich graph endpoints. * up * Add KG creation and enrichment responses * up * Remove duplicate UnstructuredChunkingConfig entry. * cleanup docs * up --------- Co-authored-by: Shreyas Pimpalgaonkar <[email protected]> * Graph docs (#1060) * fix unstr err * tweak * by_title default * cleanups * Add document chunks and enrich graph endpoints. * up * Add KG creation and enrichment responses * checkin * merge * up * Remove duplicate UnstructuredChunkingConfig entry. * Remove unused kg_search settings. * Refactor knowledge graph settings handling. * Update image and clean up logs. --------- Co-authored-by: emrgnt-cmplxty <[email protected]> * Remove duplicate method (#1061) * update docs (#1064) * rm extra prints * fix img * Fallback logic (#1062) * fix unstr err * tweak * by_title default * cleanups * Add document chunks and enrich graph endpoints. * up * Add KG creation and enrichment responses * checkin * merge * up * Remove duplicate UnstructuredChunkingConfig entry. * Remove unused kg_search settings. * Refactor knowledge graph settings handling. * Update image and clean up logs. * Implement fallback parsing mechanism * Fallback parser * Refactor code for readability and formatting * Refactor and enhance media parsers * Update response types in router. * Remove telemetry and add logging * Refactor logging format in parsers * Refactor image and movie parsers * Fix formatting in movie_parser.py * Remove debug logging statements * Remove debug logging for chunking config * Rename debug option to build. --------- Co-authored-by: emrgnt-cmplxty <[email protected]> * Refactor response models for clarity * Refactor response types in router. * Feature/fix agent (#1065) * ready for merge * fix agent * Patch/fix 123 (#1066) * ready for merge * fix agent * fix import * Feature/add orchestration draft (#1067) * ready for merge * fix agent * fix import * Fix some of the tests (#1068) * Fix fallback parsing (#1069) * Fix fallback parsing * Fix * Compose * up * Feature/iterate on docs (#1070) * add orchestration docs * docs iteration * iterate * add images * add images * Fix restructuring enum (#1071) * Feature/formatting cleanup (#1072) * add orchestration docs * docs iteration * iterate * add images * add images * run pre-commit * reclean --------- Co-authored-by: Shreyas Pimpalgaonkar <[email protected]> Co-authored-by: Manuel R. Ciosici <[email protected]> Co-authored-by: Nolan Tremelling <[email protected]>

* Feature/orchestration v0 (#1006) * Feature/remove extra r2r abstraction (#996) * moving kg construction to enrich-graph (#984) * checkin * up * done * formatting * Feature/update ingestion issues (#985) * udpate ingestion issues * keep unbounded limit support, but default to bounded * fix * fmt * removes an unnecessary abstraction * sync changes --------- Co-authored-by: Shreyas Pimpalgaonkar <[email protected]> * first commit * move towards orchestration * tweaks * check in working ingestion * move * kg enrichment * update future, postgres compose * hatchetize ingestion pipeline * ready for prime time * finish --------- Co-authored-by: Shreyas Pimpalgaonkar <[email protected]> * Feature/add update files workflow (#1010) * add update files workflow * rm ingestion pipeline * Feature/add enrichment flow (#1013) * add update files workflow * rm ingestion pipeline * v0 restructure orch * Feature/merged enrichment flow (#1016) * add update files workflow * rm ingestion pipeline * v0 restructure orch * kg orchestration * finish kg orchestration * update service * merge * cleanups * Refactor and update GraphRAG documentation * Rm graspologic (#1034) * moving kg construction to enrich-graph (#984) * checkin * up * done * formatting * Feature/update ingestion issues (#985) * udpate ingestion issues * keep unbounded limit support, but default to bounded * fix * fmt * Add support for CharacterTextSplitter (#986) * Add support for CharacterTextSplitter Allows R2R client to override the text splitter. Example: ```python ingestion_response = client.ingest_files( file_paths=[file_path], metadatas=metadata, # optionally override chunking settings at runtime chunking_settings={ "provider": "r2r", "method": "character", "extra_fields": { "separator": "---" }, } ) ``` * fixup! Add support for CharacterTextSplitter * fixup! fixup! Add support for CharacterTextSplitter * Patch/ollama base cli (#992) * Dev (#990) * moving kg construction to enrich-graph (#984) * checkin * up * done * formatting * Feature/update ingestion issues (#985) * udpate ingestion issues * keep unbounded limit support, but default to bounded * fix * fmt * Add support for CharacterTextSplitter (#986) * Add support for CharacterTextSplitter Allows R2R client to override the text splitter. Example: ```python ingestion_response = client.ingest_files( file_paths=[file_path], metadatas=metadata, # optionally override chunking settings at runtime chunking_settings={ "provider": "r2r", "method": "character", "extra_fields": { "separator": "---" }, } ) ``` * fixup! Add support for CharacterTextSplitter * fixup! fixup! Add support for CharacterTextSplitter --------- Co-authored-by: Shreyas Pimpalgaonkar <[email protected]> Co-authored-by: Manuel R. Ciosici <[email protected]> * fix ollama cli --------- Co-authored-by: Shreyas Pimpalgaonkar <[email protected]> Co-authored-by: Manuel R. Ciosici <[email protected]> * Ingestion refactor (#991) * fix test (#993) * Increase Neo4j memory limits, add GDS plugin, and update LLM concurrency limit to 256. * Update ingestion sample file, disable KG node extraction pipe, add community processing in clustering, and enhance graph clustering queries. * Update runners (#1007) * Refactor KG clustering process to simplify community processing and enhance entity-triple retrieval from Neo4j. * Refactor Neo4j configuration for memory settings and update graph clustering logic in the KG provider. * Fix pipeline by enabling node extraction and refactor community processing logic in KGClusteringPipe. * hatchet works * throw error if you run global search before enrichment * Fix communities in local search * turn off node desc embedding * fix rag endpoint * Increase hatchet msg size * Update ingestion.py * Refactor and clean up code formatting * modified workflow * Add graph creation functionality * Refactor KG parameters and logging. * review * up --------- Co-authored-by: emrgnt-cmplxty <[email protected]> Co-authored-by: emrgnt-cmplxty <[email protected]> Co-authored-by: Manuel R. Ciosici <[email protected]> Co-authored-by: Nolan Tremelling <[email protected]> * Feature/add hatchet api key setup rebased (#1040) * add update files workflow * rm ingestion pipeline * v0 restructure orch * kg orchestration * finish kg orchestration * update service * merge * cleanups * add hatchet api key setup * cleanup * add hatchet api key setup (#1037) * add hatchet api key setup * cleanup * fix merge * cleanups * Feature/nolan logs refactored (#1041) * Update runners (#1007) * Check in logs --------- Co-authored-by: Nolan Tremelling <[email protected]> * Pull open PRs into dev (#1042) * Pull in subnet and graph PR * Add in templates * Add python files for templates in cli (#1043) * working hatchet integration (#1046) * Update local_llm_neo4j_kg.toml * Unstructured fixes (#1048) * dockerfile * Update ingestion file with new sample URL and enhance unstructured chunking configuration and error handling. * clean up * clean up dockerfile * up * Update sample file and clean code * Add hatchet-sdk dependency in project. * Update providers to include local option. * Introduce File Provider (#1044) * Draft of file provider * Some cleanup * Regenearte lock * Stream it * Use document_id as primary key * Pydantic v2 * File provider finished * Make 7272 the default port (#1045) * Fix poetry.lock * Precommit * Enhance Dockerfile and add telemetry events (#1049) * Fix File Provider (#1050) * Fix * Fix parsing pipeline * working * Feature/improve docs (#1051) * improve documentation * fix unstr * add ingestion * fix compose * Add unstructured chunking configuration updates * Revert "Add unstructured chunking configuration updates" This reverts commit bae8c0b. * Separate File Provider and Relational Database Provider (#1054) * Move to self.execute_query * Check in push * Check in * Get file provider running * Actually use file provider * Final touches * undo changes in compose * Patch/fix unstructured config rebased (#1059) * fix unstr err * tweak * by_title default * cleanups * checkin * merge * Graph docs (#1058) * Add document chunks and enrich graph endpoints. * up * Add KG creation and enrichment responses * up * Remove duplicate UnstructuredChunkingConfig entry. * cleanup docs * up --------- Co-authored-by: Shreyas Pimpalgaonkar <[email protected]> * Graph docs (#1060) * fix unstr err * tweak * by_title default * cleanups * Add document chunks and enrich graph endpoints. * up * Add KG creation and enrichment responses * checkin * merge * up * Remove duplicate UnstructuredChunkingConfig entry. * Remove unused kg_search settings. * Refactor knowledge graph settings handling. * Update image and clean up logs. --------- Co-authored-by: emrgnt-cmplxty <[email protected]> * Remove duplicate method (#1061) * update docs (#1064) * rm extra prints * fix img * Fallback logic (#1062) * fix unstr err * tweak * by_title default * cleanups * Add document chunks and enrich graph endpoints. * up * Add KG creation and enrichment responses * checkin * merge * up * Remove duplicate UnstructuredChunkingConfig entry. * Remove unused kg_search settings. * Refactor knowledge graph settings handling. * Update image and clean up logs. * Implement fallback parsing mechanism * Fallback parser * Refactor code for readability and formatting * Refactor and enhance media parsers * Update response types in router. * Remove telemetry and add logging * Refactor logging format in parsers * Refactor image and movie parsers * Fix formatting in movie_parser.py * Remove debug logging statements * Remove debug logging for chunking config * Rename debug option to build. --------- Co-authored-by: emrgnt-cmplxty <[email protected]> * Refactor response models for clarity * Refactor response types in router. * Feature/fix agent (#1065) * ready for merge * fix agent * Patch/fix 123 (#1066) * ready for merge * fix agent * fix import * Feature/add orchestration draft (#1067) * ready for merge * fix agent * fix import * bump (#1075) * Enhance KG search capabilities and examples. * Fix formatting and update documentation. * Remove debug print statements in parsing. * Fix content and search level values * Update OpenAPI spec and responses. * Add customizable RAG assistant example * Update API documentation and search models. --------- Co-authored-by: emrgnt-cmplxty <[email protected]> Co-authored-by: emrgnt-cmplxty <[email protected]> Co-authored-by: Manuel R. Ciosici <[email protected]> Co-authored-by: Nolan Tremelling <[email protected]>

emrgnt-cmplxty added 2 commits September 4, 2024 17:08

improve documentation

8512200

merge doc changes

5e1c9e6

emrgnt-cmplxty changed the base branch from main to dev September 5, 2024 00:30

emrgnt-cmplxty marked this pull request as ready for review September 5, 2024 00:30

mintlify bot deployed to staging September 5, 2024 00:31 View deployment

ellipsis-dev bot reviewed Sep 5, 2024

View reviewed changes

emrgnt-cmplxty added 2 commits September 4, 2024 18:56

fix unstr

e8575e5

add ingestion

7d9564e

ellipsis-dev bot reviewed Sep 5, 2024

View reviewed changes

emrgnt-cmplxty merged commit bdb9607 into dev Sep 5, 2024
5 of 11 checks passed

emrgnt-cmplxty deleted the feature/improve-docs branch September 6, 2024 18:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Feature/improve docs #1051

Feature/improve docs #1051

Uh oh!

emrgnt-cmplxty commented Sep 5, 2024 •

edited by ellipsis-dev bot

Loading

Uh oh!

ellipsis-dev bot left a comment

Uh oh!

ellipsis-dev bot Sep 5, 2024

Uh oh!

ellipsis-dev bot Sep 5, 2024

Uh oh!

vercel bot commented Sep 5, 2024

Uh oh!

ellipsis-dev bot left a comment

Uh oh!

Uh oh!

Uh oh!

Feature/improve docs #1051

Feature/improve docs #1051

Uh oh!

Conversation

emrgnt-cmplxty commented Sep 5, 2024 • edited by ellipsis-dev bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary:

Uh oh!

ellipsis-dev bot left a comment

Choose a reason for hiding this comment

Uh oh!

ellipsis-dev bot Sep 5, 2024

Choose a reason for hiding this comment

Uh oh!

ellipsis-dev bot Sep 5, 2024

Choose a reason for hiding this comment

Uh oh!

vercel bot commented Sep 5, 2024

Uh oh!

ellipsis-dev bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

emrgnt-cmplxty commented Sep 5, 2024 •

edited by ellipsis-dev bot

Loading