Document Processor v2 (#442)

* wip: init refactor of document processor to JS

* add NodeJs PDF support

* wip: partity with python processor
feat: add pptx support

* fix: forgot files

* Remove python scripts totally

* wip:update docker to boot new collector

* add package.json support

* update dockerfile for new build

* update gitignore and linting

* add more protections on file lookup

* update package.json

* test build

* update docker commands to use cap-add=SYS_ADMIN so web scraper can run
update all scripts to reflect this
remove docker build for branch

Author: timothycarambat

README.md

@@ -74,10 +74,10 @@ Some cool features of AnythingLLM
 
 ### Technical Overview
 This monorepo consists of three main sections:
-- `collector`: Python tools that enable you to quickly convert online resources or local documents into LLM useable format.
 - `frontend`: A viteJS + React frontend that you can run to easily create and manage all your content the LLM can use.
-- `server`: A nodeJS + express server to handle all the interactions and do all the vectorDB management and LLM interactions.
+- `server`: A NodeJS express server to handle all the interactions and do all the vectorDB management and LLM interactions.
 - `docker`: Docker instructions and build process + information for building from source.
+- `collector`: NodeJS express server that process and parses documents from the UI.
 
 ### Minimum Requirements
 > [!TIP]
@@ -86,7 +86,6 @@ This monorepo consists of three main sections:
 > you will be storing (documents, vectors, models, etc). Minimum 10GB recommended.
 
 - `yarn` and `node` on your machine
-- `python` 3.9+ for running scripts in `collector/`.
 - access to an LLM running locally or remotely.
 
 *AnythingLLM by default uses a built-in vector database powered by [LanceDB](https://github.com/lancedb/lancedb)
@@ -112,6 +111,7 @@ export STORAGE_LOCATION="/var/lib/anythingllm" && \
 mkdir -p $STORAGE_LOCATION && \
 touch "$STORAGE_LOCATION/.env" && \
 docker run -d -p 3001:3001 \
+--cap-add SYS_ADMIN \
 -v ${STORAGE_LOCATION}:/app/server/storage \
 -v ${STORAGE_LOCATION}/.env:/app/server/.env \
 -e STORAGE_DIR="/app/server/storage" \
@@ -141,12 +141,6 @@ To boot the frontend locally (run commands from root of repo):
 
 [Learn about vector caching](./server/storage/vector-cache/VECTOR_CACHE.md)
 
-## Standalone scripts
-
-This repo contains standalone scripts you can run to collect data from a Youtube Channel, Medium articles, local text files, word documents, and the list goes on. This is where you will use the `collector/` part of the repo.
-
-[Go set up and run collector scripts](./collector/README.md)
-
 ## Contributing
 - create issue
 - create PR with branch name format of `<issue number>-<short name>`

cloud-deployments/aws/cloudformation/DEPLOY.md

@@ -1,6 +1,6 @@
 # How to deploy a private AnythingLLM instance on AWS
 
-With an AWS account you can easily deploy a private AnythingLLM instance on AWS. This will create a url that you can access from any browser over HTTP (HTTPS not supported). This single instance will run on your own keys and they will not be exposed - however if you want your instance to be protected it is highly recommend that you set the `AUTH_TOKEN` and `JWT_SECRET` variables in the `docker/` ENV.
+With an AWS account you can easily deploy a private AnythingLLM instance on AWS. This will create a url that you can access from any browser over HTTP (HTTPS not supported). This single instance will run on your own keys and they will not be exposed - however if you want your instance to be protected it is highly recommend that you set a password one setup is complete.
 
 **Quick Launch (EASY)**
 1. Log in to your AWS account
@@ -30,12 +30,11 @@ The output of this cloudformation stack will be:
 
 **Requirements**
 - An AWS account with billing information.
-  - AnythingLLM (GUI + document processor) must use a t2.small minimum and 10Gib SSD hard disk volume
 
 ## Please read this notice before submitting issues about your deployment
 
 **Note:** 
-Your instance will not be available instantly. Depending on the instance size you launched with it can take varying amounts of time to fully boot up.
+Your instance will not be available instantly. Depending on the instance size you launched with it can take 5-10 minutes to fully boot up.
 
 If you want to check the instance's progress, navigate to [your deployed EC2 instances](https://us-west-1.console.aws.amazon.com/ec2/home) and connect to your instance via SSH in browser.
 

cloud-deployments/aws/cloudformation/cloudformation_create_anythingllm.json

@@ -89,7 +89,7 @@
                 "touch /home/ec2-user/anythingllm/.env\n",
                 "sudo chown ec2-user:ec2-user -R /home/ec2-user/anythingllm\n",
                 "docker pull mintplexlabs/anythingllm:master\n",
-                "docker run -d -p 3001:3001 -v /home/ec2-user/anythingllm:/app/server/storage -v /home/ec2-user/anythingllm/.env:/app/server/.env -e STORAGE_DIR=\"/app/server/storage\" mintplexlabs/anythingllm:master\n",
+                "docker run -d -p 3001:3001 --cap-add SYS_ADMIN -v /home/ec2-user/anythingllm:/app/server/storage -v /home/ec2-user/anythingllm/.env:/app/server/.env -e STORAGE_DIR=\"/app/server/storage\" mintplexlabs/anythingllm:master\n",
                 "echo \"Container ID: $(sudo docker ps --latest --quiet)\"\n",
                 "export ONLINE=$(curl -Is http://localhost:3001/api/ping | head -n 1|cut -d$' ' -f2)\n",
                 "echo \"Health check: $ONLINE\"\n",

cloud-deployments/digitalocean/terraform/DEPLOY.md

@@ -1,8 +1,6 @@
 # How to deploy a private AnythingLLM instance on DigitalOcean using Terraform
 
-With a DigitalOcean account, you can easily deploy a private AnythingLLM instance using Terraform. This will create a URL that you can access from any browser over HTTP (HTTPS not supported). This single instance will run on your own keys, and they will not be exposed. However, if you want your instance to be protected, it is highly recommended that you set the `AUTH_TOKEN` and `JWT_SECRET` variables in the `docker/` ENV.
-
-[Refer to .env.example](../../../docker/HOW_TO_USE_DOCKER.md) for data format.
+With a DigitalOcean account, you can easily deploy a private AnythingLLM instance using Terraform. This will create a URL that you can access from any browser over HTTP (HTTPS not supported). This single instance will run on your own keys, and they will not be exposed. However, if you want your instance to be protected, it is highly recommended that you set a password one setup is complete.
 
 The output of this Terraform configuration will be:
 - 1 DigitalOcean Droplet
@@ -12,8 +10,6 @@ The output of this Terraform configuration will be:
 - An DigitalOcean  account with billing information
 - Terraform installed on your local machine
   - Follow the instructions in the [official Terraform documentation](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli) for your operating system.
-- `.env` file that is filled out with your settings and set up in the `docker/` folder
-
 
 ## How to deploy on DigitalOcean
 Open your terminal and navigate to the `digitalocean/terraform` folder
@@ -36,7 +32,7 @@ terraform destroy
 ## Please read this notice before submitting issues about your deployment
 
 **Note:** 
-Your instance will not be available instantly. Depending on the instance size you launched with it can take anywhere from 10-20 minutes to fully boot up.
+Your instance will not be available instantly. Depending on the instance size you launched with it can take anywhere from 5-10 minutes to fully boot up.
 
 If you want to check the instances progress, navigate to [your deployed instances](https://cloud.digitalocean.com/droplets) and connect to your instance via SSH in browser.
 

cloud-deployments/digitalocean/terraform/user_data.tp1

@@ -12,7 +12,7 @@ mkdir -p /home/anythingllm
 touch /home/anythingllm/.env
 
 sudo docker pull mintplexlabs/anythingllm:master
-sudo docker run -d -p 3001:3001 -v /home/anythingllm:/app/server/storage -v /home/anythingllm/.env:/app/server/.env -e STORAGE_DIR="/app/server/storage" mintplexlabs/anythingllm:master
+sudo docker run -d -p 3001:3001 --cap-add SYS_ADMIN -v /home/anythingllm:/app/server/storage -v /home/anythingllm/.env:/app/server/.env -e STORAGE_DIR="/app/server/storage" mintplexlabs/anythingllm:master
 echo "Container ID: $(sudo docker ps --latest --quiet)"  
 
 export ONLINE=$(curl -Is http://localhost:3001/api/ping | head -n 1|cut -d$' ' -f2)  

cloud-deployments/gcp/deployment/DEPLOY.md

@@ -1,8 +1,6 @@
 # How to deploy a private AnythingLLM instance on GCP
 
-With a GCP account you can easily deploy a private AnythingLLM instance on GCP. This will create a url that you can access from any browser over HTTP (HTTPS not supported). This single instance will run on your own keys and they will not be exposed - however if you want your instance to be protected it is highly recommend that you set the `AUTH_TOKEN` and `JWT_SECRET` variables in the `docker/` ENV.
-
-[Refer to .env.example](../../../docker/HOW_TO_USE_DOCKER.md) for data format.
+With a GCP account you can easily deploy a private AnythingLLM instance on GCP. This will create a url that you can access from any browser over HTTP (HTTPS not supported). This single instance will run on your own keys and they will not be exposed - however if you want your instance to be protected it is highly recommend that you set a password one setup is complete.
 
 The output of this cloudformation stack will be:
 - 1 GCP VM
@@ -11,19 +9,15 @@ The output of this cloudformation stack will be:
 
 **Requirements**
 - An GCP account with billing information.
-  - AnythingLLM (GUI + document processor) must use a n1-standard-1 minimum and 10Gib SSD hard disk volume
-- `.env` file that is filled out with your settings and set up in the `docker/` folder
 
 ## How to deploy on GCP
 Open your terminal
-1. Generate your specific cloudformation document by running `yarn generate:gcp_deployment` from the project root directory.
-2. This will create a new file (`gcp_deploy_anything_llm_with_env.yaml`) in the `gcp/deployment` folder.
-3. Log in to your GCP account using the following command:
+1. Log in to your GCP account using the following command:
     ```
     gcloud auth login 
     ```
 
-4. After successful login, Run the following command to create a deployment using the Deployment Manager CLI:
+2. After successful login, Run the following command to create a deployment using the Deployment Manager CLI:
 
   ```
 
@@ -57,5 +51,4 @@ If you want to check the instances progress, navigate to [your deployed instance
 
 Once connected run `sudo tail -f /var/log/cloud-init-output.log` and wait for the file to conclude deployment of the docker image.
 
-
-Additionally, your use of this deployment process means you are responsible for any costs of these GCP resources fully.
+Additionally, your use of this deployment process means you are responsible for any costs of these GCP resources fully.

cloud-deployments/gcp/deployment/gcp_deploy_anything_llm.yaml

@@ -34,7 +34,7 @@ resources:
               touch /home/anythingllm/.env
                
               sudo docker pull mintplexlabs/anythingllm:master
-              sudo docker run -d -p 3001:3001 -v /home/anythingllm:/app/server/storage -v /home/anythingllm/.env:/app/server/.env -e STORAGE_DIR="/app/server/storage" mintplexlabs/anythingllm:master
+              sudo docker run -d -p 3001:3001 --cap-add SYS_ADMIN -v /home/anythingllm:/app/server/storage -v /home/anythingllm/.env:/app/server/.env -e STORAGE_DIR="/app/server/storage" mintplexlabs/anythingllm:master
               echo "Container ID: $(sudo docker ps --latest --quiet)"  
 
               export ONLINE=$(curl -Is http://localhost:3001/api/ping | head -n 1|cut -d$' ' -f2)  

cloud-deployments/gcp/deployment/generate.mjs

@@ -1,8 +1,6 @@
-outputs/*/*.json
 hotdir/*
-hotdir/processed/*
-hotdir/failed/*
 !hotdir/__HOTDIR__.md
-!hotdir/processed
-!hotdir/failed
-
+yarn-error.log
+!yarn.lock
+outputs
+scripts

collector/.env.example

@@ -0,0 +1 @@
+v18.13.0

collector/.gitignore

@@ -1,17 +1,3 @@
 ### What is the "Hot directory"
 
-This is the location where you can dump all supported file types and have them automatically converted and prepared to be digested by the vectorizing service and selected from the AnythingLLM frontend.
-
-Files dropped in here will only be processed when you are running `python watch.py` from the `collector` directory.
-
-Once converted the original file will be moved to the `hotdir/processed` folder so that the original document is still able to be linked to when referenced when attached as a source document during chatting.
-
-**Supported File types**
-- `.md`
-- `.txt`
-- `.pdf`
-
-__requires more development__
-- `.png .jpg etc`
-- `.mp3`
-- `.mp4`
+This is a pre-set file location that documents will be written to when uploaded by AnythingLLM. There is really no need to touch it.