run_llm_demo.md

Running the LLM App Demo

Before you begin

1. Make sure you've setup and initialized your Database.

2. Make sure you've deployed your retrieval service, and are running a connection to it locally on 127.0.0.1:8080.

3. Make sure you have Python 3.11+ installed

Setting up your Environment

1. Set up Application Default Credentials:

   gcloud auth application-default login
   • Tip: if you are running into 403 error, check to make sure the service account you are using has the Cloud Run Invoker IAM in the retrieval service project.

2. Change into the llm_demo directory:

   cd llm_demo 

3. Set orchestrator environment variable:

   orchestration-type	Description
   langchain-tools	LangChain tools orchestrator.
   vertexai-function-calling	VertexAI Function Calling orchestrator.

   export ORCHESTRATION_TYPE=<orchestration-type>

4. Install the dependencies using pip. You may wish to do this in a venv:

   pip install -r requirements.txt

5. [Optional] If you want to take advantage of the user authentication features, create a Client ID for your app and save it as an environment variable:

   export CLIENT_ID=<Your Client ID>

6. [Optional] To make session information readable but not modifiable, you can customized the signed cookie-based HTTP sessions. Defaulted to "this is a secret". More information available here.

   export MIDDLEWARE_SECRET=<random string>

Running the Demo

1. Start the application with:

   python run_app.py

   Note: for hot reloading of the app use: python run_app.py --reload

2. View app at http://localhost:8081/