Merge branch 'master' of github.com:Mintplex-Labs/anything-llm into r…

…ender

.devcontainer/README.md

@@ -41,11 +41,12 @@ Checklist:
 
 :warning: **Important for all developers** :warning:
 
-- [ ] Whe you are using the `NODE_ENV=development` the server will not store the configurations you set for security reasons. Please set the proper config on file `.env.development`. The side-effect if you don't, everytime you restart the server, you will be sent to the "Onboarding" page again.
+- [ ] When you are using the `NODE_ENV=development` the server will not store the configurations you set for security reasons. Please set the proper config on file `.env.development`. The side-effect if you don't, everytime you restart the server, you will be sent to the "Onboarding" page again.
 
-:warning: **Important for Github Codespaces** :warning:
+**Note when using Github Codespaces**
+
+- [ ] When running the "Server" for the first time, it will automatically configure its port to be publicly accessible by default, as this is required for the front end to reach the server backend. To know more, read the content of the `.env` file on the frontend folder about this, and if any issues occur, make sure to manually set the port "Visibility" of the "Server" is set to "Public" if needed. Again, this is only needed for developing on Github Codespaces.
 
-- [ ] When running the "Server" for the first time, its port will be automatically forward, but privately. Read the content of the `.env` file on the frontend folder about this, and make sure the port "Visibility" is set to "Public", so the frontend can reach the backend. Again, this is only needed for developing on Github Codespaces. We appreciate to know if you have a better solution.
 
 **For the Collector:**
 

.github/workflows/build-and-push-image.yaml

@@ -21,7 +21,8 @@ on:
       - '**/.env.example'
       - '.github/ISSUE_TEMPLATE/**/*'
       - '.devcontainer/**/*'
-      - 'embed/**/*' # Embed should be published to frontend (yarn build:publish) if any changes are introduced
+      - 'embed/**/*' # Embed is submodule
+      - 'browser-extension/**/*' # Chrome extension is submodule
       - 'server/utils/agents/aibitat/example/**/*' # Do not push new image for local dev testing of new aibitat images.
 
 jobs:

.github/workflows/dev-build.yaml

@@ -6,7 +6,7 @@ concurrency:
 
 on:
   push:
-    branches: ['encrypt-jwt-value'] # put your current branch to create a build. Core team only.
+    branches: ['agent-skill-plugins'] # put your current branch to create a build. Core team only.
     paths-ignore:
       - '**.md'
       - 'cloud-deployments/*'

.gitmodules

@@ -0,0 +1,7 @@
+[submodule "browser-extension"]
+	path = browser-extension
+	url = [email protected]:Mintplex-Labs/anythingllm-extension.git
+[submodule "embed"]
+	path = embed
+	url = [email protected]:Mintplex-Labs/anythingllm-embed.git
+	branch = main

.vscode/settings.json

@@ -41,8 +41,10 @@
     "Qdrant",
     "royalblue",
     "searxng",
+    "SearchApi",
     "Serper",
     "Serply",
+    "streamable",
     "textgenwebui",
     "togetherai",
     "Unembed",

README.md

@@ -69,7 +69,7 @@ AnythingLLM divides your documents into objects called `workspaces`. A Workspace
 
 ### Supported LLMs, Embedder Models, Speech models, and Vector Databases
 
-**Language Learning Models:**
+**Large Language Models (LLMs):**
 
 - [Any open-source llama.cpp compatible model](/server/storage/models/README.md#text-generation-llm-selection)
 - [OpenAI](https://openai.com)
@@ -137,7 +137,8 @@ This monorepo consists of three main sections:
 - `server`: A NodeJS express server to handle all the interactions and do all the vectorDB management and LLM interactions.
 - `collector`: NodeJS express server that process and parses documents from the UI.
 - `docker`: Docker instructions and build process + information for building from source.
-- `embed`: Code specifically for generation of the [embed widget](./embed/README.md).
+- `embed`: Submodule for generation & creation of the [web embed widget](https://github.com/Mintplex-Labs/anythingllm-embed).
+- `browser-extension`: Submodule for the [chrome browser extension](https://github.com/Mintplex-Labs/anythingllm-extension).
 
 ## 🛳 Self Hosting
 
@@ -146,9 +147,9 @@ Mintplex Labs & the community maintain a number of deployment methods, scripts,
 |----------------------------------------|----:|-----|---------------|------------|
 | [![Deploy on Docker][docker-btn]][docker-deploy] | [![Deploy on AWS][aws-btn]][aws-deploy] | [![Deploy on GCP][gcp-btn]][gcp-deploy] | [![Deploy on DigitalOcean][do-btn]][do-deploy] | [![Deploy on Render.com][render-btn]][render-deploy] |
 
-| Railway  |  RepoCloud |
-| --- | --- |
-| [![Deploy on Railway][railway-btn]][railway-deploy] | [![Deploy on RepoCloud][repocloud-btn]][repocloud-deploy] |
+| Railway  |  RepoCloud | Elestio |
+| --- | --- | --- |
+| [![Deploy on Railway][railway-btn]][railway-deploy] | [![Deploy on RepoCloud][repocloud-btn]][repocloud-deploy] | [![Deploy on Elestio][elestio-btn]][elestio-deploy] |
 
 [or set up a production AnythingLLM instance without Docker →](./BARE_METAL.md)
 
@@ -246,3 +247,5 @@ This project is [MIT](./LICENSE) licensed.
 [railway-deploy]: https://railway.app/template/HNSCS1?referralCode=WFgJkn
 [repocloud-btn]: https://d16t0pc4846x52.cloudfront.net/deploylobe.svg
 [repocloud-deploy]: https://repocloud.io/details/?app_id=276
+[elestio-btn]: https://elest.io/images/logos/deploy-to-elestio-btn.png
+[elestio-deploy]: https://elest.io/open-source/anythingllm

collector/extensions/index.js

@@ -38,7 +38,6 @@ function extensions(app) {
           reqBody(request),
           response,
         );
-        console.log({ success, reason, data })
         response.status(200).json({
           success,
           reason,

collector/utils/extensions/RepoLoader/GithubRepo/RepoLoader/index.js

@@ -119,6 +119,7 @@ class GitHubRepoLoader {
       maxConcurrency: 5,
       unknown: "ignore",
       ignorePaths: this.ignorePaths,
+      verbose: true,
     });
 
     const docs = [];

docker/.env.example

@@ -252,6 +252,10 @@ GID='1000'
 # AGENT_GSE_KEY=
 # AGENT_GSE_CTX=
 
+#------ SearchApi.io ----------- https://www.searchapi.io/
+# AGENT_SEARCHAPI_API_KEY=
+# AGENT_SEARCHAPI_ENGINE=google
+
 #------ Serper.dev ----------- https://serper.dev/
 # AGENT_SERPER_DEV_KEY=
 

docker/HOW_TO_USE_DOCKER.md

@@ -117,8 +117,8 @@ services:
       - WHISPER_PROVIDER=local
       - TTS_PROVIDER=native
       - PASSWORDMINCHAR=8
-      - AGENT_SERPER_DEV_KEY="SERPER DEV API KEY"
-      - AGENT_SERPLY_API_KEY="Serply.io API KEY"
+      # Add any other keys here for services or settings
+      # you can find in the docker/.env.example file
     volumes:
       - anythingllm_storage:/app/server/storage
     restart: always