From 04c5a742aefb070633eedf65a976df9479dfba96 Mon Sep 17 00:00:00 2001 From: Raz Rotenberg Date: Sun, 18 Feb 2024 12:50:01 +0200 Subject: [PATCH] Documented LLMs overview - Added missing content index to remote overview --- docs/source/index.rst | 6 ++ docs/source/llms/overview.rst | 126 ++++++++++++++++++++++++++++++ docs/source/overview/overview.rst | 3 + docs/source/remote/overview.rst | 4 + 4 files changed, 139 insertions(+) create mode 100644 docs/source/llms/overview.rst diff --git a/docs/source/index.rst b/docs/source/index.rst index 2d6957e..1093403 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -37,6 +37,12 @@ Genv lets you easily control, configure, monitor and enforce the GPU resources t remote/enforcement remote/monitoring +.. toctree:: + :maxdepth: 2 + :caption: LLMs + + llms/overview + .. toctree:: :maxdepth: 2 :caption: Container Toolkit diff --git a/docs/source/llms/overview.rst b/docs/source/llms/overview.rst new file mode 100644 index 0000000..18d90f9 --- /dev/null +++ b/docs/source/llms/overview.rst @@ -0,0 +1,126 @@ +Overview +======== + +.. contents:: + :depth: 3 + :backlinks: none + +Genv is an environment management software for GPUs that manages GPUs on a single machine or in a cluster. + +Genv uses `Ollama `__ for managing managing Large Language Models (LLMs) with Genv. +This allows users to efficiently run, manage and utilize LLMs on GPUs within their clusters. + +This guide will help you get started with LLMs using Genv. + +Prerequisites +------------- + +First, you will need to :doc:`install <../overview/installation>` Genv on your GPU machines in the cluster. + +.. note:: + + It is recommended to install Genv :ref:`system-wide `. + +Then, you will need to `install `__ Ollama on your GPU machines with the command: + +.. code-block:: shell + + curl -fsSL https://ollama.com/install.sh | sh + +Quick start +----------- + +If you are running on a GPU cluster with multiple machines, follow the :ref:`cluster ` quick start. +Otherwise, follow the :ref:`local ` quick start. + +.. _LLM cluster quick start: + +Cluster +~~~~~~~ +In my case, I have two remote GPU machines: :code:`gpu-server-1` and :code:`gpu-server-2`. + +Serve your first LLM on a single GPU with the command: + +.. code-block:: shell + + $ genv remote -H gpu-server-1,gpu-server-2 llm serve llama2 --gpus 1 + ... + +.. note:: + + Use :code:`Ctrl+C` to stop the serving. + +Open another terminal and list all LLMs in your cluster with: + +.. code-block:: shell + + $ genv remote -H gpu-server-1,gpu-server-2 llm ps + HOST MODEL PORT CREATED EID USER + gpu-server-1 llama2 N/A 2 hours ago 571 root + +We can see that :code:`llama2` is now being served in the cluster. + +Attach to the model and interact with it using the command: + +.. code-block:: + + $ genv remote -H gpu-server-1,gpu-server-2 llm attach llama2 + >>> Hi! I'm Genv, how are you? + + Hello there, it's nice to meet you! *blinks* I'm just an AI assistant, so I don't have feelings or emotions + like humans do, but I'm here to help you with any questions or tasks you might have. How can I assist you + today? + + >>> Send a message (/? for help) + +.. note:: + + Use :code:`Ctrl+D` to exit the interactive terminal. + +.. _LLM local quick start: + +Local +~~~~~ +Serve your first LLM on a single GPU with the command: + +.. code-block:: shell + + $ genv llm serve llama2 --gpus 1 + ... + +.. note:: + + Use :code:`Ctrl+C` to stop the serving. + +Open another terminal and list all LLMs on your machine with: + +.. code-block:: shell + + $ genv llm ps + MODEL PORT CREATED EID USER PID(S) + llama2 33123 22 seconds ago 1179252 raz(1001) 1179252 + +We can see that :code:`llama2` is now being served on the machine. + +Attach to the model and interact with it using the command: + +.. code-block:: + + $ genv llm attach llama2 + >>> Hello from Genv + + Hello there! *blinks* Uh, I'm... uh... *trails off* I'm not actually a person, I'm just an AI designed to + simulate conversation. So, I don't have feelings or emotions like a real person would. But I'm here to help + you with any questions or tasks you may have! How can I assist you today? + + >>> Send a message (/? for help) + +.. note:: + + Use :code:`Ctrl+D` to exit the interactive terminal. + +Where to Go Next +---------------- +If you are not familiar with Genv, it is recommended to go over the :doc:`overview <../overview/overview>` and follow the :doc:`quick start tutorial <../overview/quickstart>`. + +You should also learn how :doc:`remote features <../remote/overview>` work in Genv. diff --git a/docs/source/overview/overview.rst b/docs/source/overview/overview.rst index fdea61a..d91e037 100644 --- a/docs/source/overview/overview.rst +++ b/docs/source/overview/overview.rst @@ -18,6 +18,8 @@ If you are not familiar with the concept of environments, you can check out the Over time, the project developed into a complete GPU cluster management tool with various capabilities and mechanisms like :doc:`monitoring <../usage/monitoring>`, :doc:`enforcement <../usage/enforcement>`, :doc:`remote capabilities <../remote/overview>` and more. +In addition to running GPU environments in a cluster, Genv also supports managing :doc:`LLMs <../llms/overview>` on GPUs. + At its core, Genv is a `Python package `__ that provisions GPUs to environments and keeps its state as :ref:`files ` at :code:`/var/tmp/genv`. On top of this core layer, Genv has a CLI (i.e. the command :code:`genv`), :doc:`Python SDK <../usage/python-sdk>` and :ref:`integrations ` with many common tools and environments such as terminal, containers (e.g. :code:`docker`), Ray, Visual Studio Code, JupyterLab and PyCharm. @@ -55,5 +57,6 @@ For more information, see :doc:`here <../remote/overview>`. Where to Go Next ---------------- If you are not familiar with Genv, it is recommended to follow the :doc:`quick start tutorial `. +If you are into LLMs, check out the :doc:`LLMs overview <../llms/overview>`. If you are looking for how to install Genv, visit the :doc:`installation page `, and if you want to know how to do specific things, check out the :doc:`usage guide <../usage/usage>`. diff --git a/docs/source/remote/overview.rst b/docs/source/remote/overview.rst index ecb7759..56db025 100644 --- a/docs/source/remote/overview.rst +++ b/docs/source/remote/overview.rst @@ -1,6 +1,10 @@ Overview ======== +.. contents:: + :depth: 3 + :backlinks: none + Genv is an environment management software for GPUs. It manages all GPUs on a single machine.