Properly organize documentation instead of sticking it all in the README

docs/advanced_usage.rst

@@ -0,0 +1,163 @@
+Advanced Usage
+==============
+
+Zero-Downtime Restarts
+----------------------
+
+Prior to Gravity 1.0, the preferred solution for performing zero-downtime restarts was `unicornherder`_. However, due to
+limitations in the unicornherder software, it does not always successfully perform zero-downtime restarts. Because of
+this, Gravity is now able to perform rolling restarts of gunicorn services if more than one gunicorn is configured.
+
+To run multiple gunicorn processes, configure the ``gunicorn`` section of the Gravity configuration as a *list*. Each
+item in the list is a gunicorn configuration, and can have all of the same parameters as a single gunicorn
+configuration:
+
+.. code:: yaml
+
+    gravity:
+      gunicorn:
+        - bind: unix:/srv/galaxy/var/gunicorn0.sock
+          workers: 4
+        - bind: unix:/srv/galaxy/var/gunicorn1.sock
+          workers: 4
+
+.. caution::
+
+   This will start multiple Galaxy servers with the same ``server_name``. If you have not configured separate Galaxy
+   processes to act as job handlers, your gunicorn processes will handle them, resulting in job errors due to handling
+   the same job multiple times. See the Gravity and Galaxy documentation on configuring handlers.
+
+Your proxy server can balance load between the two gunicorns. For example, with nginx:
+
+.. code:: nginx
+
+    upstream galaxy {
+        server unix:/srv/galaxy/var/gunicorn0.sock;
+        server unix:/srv/galaxy/var/gunicorn1.sock;
+    }
+
+    http {
+        location / {
+            proxy_pass http://galaxy;
+        }
+    }
+
+By default, Gravity will wait 300 seconds for the gunicorn server to respond to web requests after initiating the
+restart. To change this timeout this, set the ``restart_timeout`` option on each configured ``gunicorn`` instance.
+
+Service Instances
+-----------------
+
+In the case of multiple gunicorn instances as described in :ref:`Zero-Downtime Restarts` and multiple dynamic handlers
+as described in :ref:`Galaxy Job Handlers`, Gravity will create multiple *service instances* of each service. This
+allows multiple processes to be run from a single service definition.
+
+In supervisor, this means that the service names as presented by supervisor are appended with ``:INSTANCE_NUMBER``,
+e.g.:
+
+.. code:: console
+
+    $ galaxyctl status
+    celery                           RUNNING   pid 121363, uptime 0:02:33
+    celery-beat                      RUNNING   pid 121364, uptime 0:02:33
+    gunicorn:0                       RUNNING   pid 121365, uptime 0:02:33
+    gunicorn:1                       RUNNING   pid 121366, uptime 0:02:33
+
+However, ``galaxyctl`` commands that take a service name still use the base service name, e.g.:
+
+.. code:: console
+
+    $ galaxyctl stop gunicorn
+    gunicorn:0: stopped
+    gunicorn:1: stopped
+    Not all processes stopped, supervisord not shut down (hint: see `galaxyctl status`)
+
+In systemd, the service names as presented by systemd are appended with ``@INSTANCE_NUMBER``,
+e.g.:
+
+.. code:: console
+
+    $ galaxyctl status
+      UNIT                       LOAD   ACTIVE SUB     DESCRIPTION
+      galaxy-celery-beat.service loaded active running Galaxy celery-beat
+      galaxy-celery.service      loaded active running Galaxy celery
+      [email protected]  loaded active running Galaxy gunicorn (process 0)
+      [email protected]  loaded active running Galaxy gunicorn (process 1)
+      galaxy.target              loaded active active  Galaxy
+
+As with supervisor, ``galaxyctl`` commands that take a service name still use the base service name.
+
+If you prefer not to work with service instances and want Galaxy to write a service configuration file for each instance
+of each service, you can do so by setting ``use_service_instances`` in the Gravity configuration to ``false``.
+
+Managing Multiple Galaxies
+--------------------------
+
+Gravity can manage multiple instances of Galaxy simultaneously. This is useful especially in the case where you have
+multiple production Galaxy instances on a single server and are managing them with Gravity installed outside of a Galaxy
+virtualenv, as root. There are multiple ways to achieve this:
+
+1. Pass multiple ``--config-file`` options to ``galaxyctl``, or set a list of colon-separated config paths in
+   ``$GRAVITY_CONFIG_FILE``:
+
+    .. code:: console
+
+        $ galaxyctl --config-file /srv/galaxy/test/config/galaxy.yml \
+                    --config-file /srv/galaxy/main/config/galaxy.yml list --version
+        TYPE      INSTANCE NAME       VERSION       CONFIG PATH
+        galaxy    test                22.05         /srv/galaxy/test/config/galaxy.yml
+        galaxy    main                22.09.dev0    /srv/galaxy/main/config/galaxy.yml
+        $ export GRAVITY_CONFIG_FILE='/srv/galaxy/test/config/galaxy.yml:/srv/galaxy/main/config/galaxy.yml'
+        $ galaxyctl list --version
+        TYPE      INSTANCE NAME       VERSION       CONFIG PATH
+        galaxy    test                22.05         /srv/galaxy/test/config/galaxy.yml
+        galaxy    main                22.09.dev0    /srv/galaxy/main/config/galaxy.yml
+
+2. If running as root, any config files located in ``/etc/galaxy/gravity.d`` will automatically be loaded.
+
+3. Specify multiple Gravity configurations in a single config file, as a list. In this case, the Galaxy and Gravity
+   configurations must be in separate files as described in :ref:`Splitting Gravity and Galaxy Configurations`:
+
+    .. code:: yaml
+
+        gravity:
+          - instance_name: test
+            process_manager: systemd
+            galaxy_config_file: /srv/galaxy/test/config/galaxy.yml
+            galaxy_root: /srv/galaxy/test/server
+            virtualenv: /srv/galaxy/test/venv
+            galaxy_user: gxtest
+            gunicorn:
+              bind: unix:/srv/galaxy/test/var/gunicorn.sock
+            handlers:
+              handler:
+                pools:
+                  - job-handlers
+                  - workflow-schedulers
+
+          - instance_name: main
+            process_manager: systemd
+            galaxy_config_file: /srv/galaxy/main/config/galaxy.yml
+            galaxy_root: /srv/galaxy/main/server
+            virtualenv: /srv/galaxy/main/venv
+            galaxy_user: gxmain
+            gunicorn:
+              bind: unix:/srv/galaxy/main/var/gunicorn.sock
+              workers: 8
+            handlers:
+              handler:
+                processes: 4
+                pools:
+                  - job-handlers
+                  - workflow-schedulers
+
+In all cases, when using multiple Gravity instances, each Galaxy instance managed by Gravity must have a unique
+**instance name**. When working with a single instance, the default name ``_default_`` is used automatically and mostly
+hidden from you. When working with multiple instances, set the ``instance_name`` option in each instance's Gravity
+config to a unique name.
+
+Although it is strongly encouraged to use systemd for running multiple instances, it is possible to use supervisor.
+Please see the :ref:`Gravity State` section for important details on how and where Gravity stores the supervisor
+configuration and log files.
+
+.. _unicornherder: https://github.com/alphagov/unicornherder

docs/basic_usage.rst

@@ -0,0 +1,176 @@
+Basic Usage
+===========
+
+A basic example of starting and running a simple Galaxy server from a source clone in the foreground is provided in the
+ref:`Quick Start` guide. This section covers more typical usage for production Galaxy servers.
+
+Managing a Single Galaxy
+------------------------
+
+If you have not installed Gravity separate from the Galaxy virtualenv, simply activate Galaxy's virtualenv, which will
+put Gravity's ``galaxyctl`` and ``galaxy`` commands on your ``$PATH``:
+
+.. code:: console
+
+    $ . /srv/galaxy/venv/bin/activate
+    $ galaxyctl --help
+    Usage: galaxyctl [OPTIONS] COMMAND [ARGS]...
+
+      Manage Galaxy server configurations and processes.
+
+    Options:
+      -d, --debug             Enables debug mode.
+      -c, --config-file FILE  Gravity (or Galaxy) config file to operate on. Can
+                              also be set with $GRAVITY_CONFIG_FILE or
+                              $GALAXY_CONFIG_FILE
+      --state-dir DIRECTORY   Where process management configs and state will be
+                              stored.
+      -h, --help              Show this message and exit.
+
+    Commands:
+      configs     List registered config files.
+      deregister  Deregister config file(s).
+      exec        Run a single Galaxy service in the foreground, with logging...
+      follow      Follow log files of configured instances.
+      graceful    Gracefully reload configured services.
+      instances   List all known instances.
+      pm          Invoke process manager (supervisorctl, systemctl) directly.
+      register    Register config file(s).
+      rename      Update path of registered config file.
+      restart     Restart configured services.
+      show        Show details of registered config.
+      shutdown    Shut down process manager.
+      start       Start configured services.
+      status      Display server status.
+      stop        Stop configured services.
+      update      Update process manager from config changes.
+
+If you run ``galaxy`` or ``galaxyctl`` from the root of a Galaxy source checkout and do not specify the config file
+option, ``config/galaxy.yml`` or ``config/galaxy.yml.sample`` will be automatically used. This is handy for working with
+local clones of Galaxy for testing or development. You can skip Galaxy's lengthy and repetitive ``run.sh`` configuration
+steps when starting and stopping Galaxy in between code updates (you should still run ``run.sh`` after performing a
+``git pull`` to make sure your dependencies are up to date).
+
+Gravity can either run Galaxy via the `supervisor`_ process manager (the default) or `systemd`_. For production servers,
+**it is recommended that you run Gravity as root in systemd mode**. See the :ref:`Managing a Production Galaxy` section
+for details.
+
+As shown in the Quick Start, the ``galaxy`` command will run a Galaxy server in the foreground. The ``galaxy`` command
+is actually a shortcut for two separate steps: 1. read the provided ``galaxy.yml`` and write out the corresponding
+process manager configurations, and 2. start and run Galaxy in the foreground using the process manager (`supervisor`_).
+You can perform these steps separately (and in this example, start Galaxy as a backgrounded daemon instead of in the
+foreground):
+
+.. code:: console
+
+    $ galaxyctl update
+    Adding service gunicorn
+    Adding service celery
+    Adding service celery-beat
+    $ galaxyctl start
+    celery                           STARTING
+    celery-beat                      STARTING
+    gunicorn                         STARTING
+    Log files are in /srv/galaxy/var/gravity/log
+
+When running as a daemon, the ``stop`` subcommand stops your Galaxy server:
+
+.. code:: console
+
+    $ galaxyctl stop
+    celery-beat: stopped
+    gunicorn: stopped
+    celery: stopped
+    All processes stopped, supervisord will exit
+    Shut down
+
+Most Gravity subcommands (such as ``stop``, ``start``, ``restart``, ...) are straightforward, but a few subcommands are
+worth pointing out: :ref:`update`, :ref:`graceful`, and :ref:`exec`. All subcommands are documented in the
+:ref:`Subcommands` section and their respective ``--help`` output.
+
+Managing a Production Galaxy
+----------------------------
+
+By default, Gravity runs Galaxy processes under `supervisor`_, but setting the ``process_manager`` option to ``systemd``
+in Gravity's configuration will cause it to run under `systemd`_ instead. systemd is the default init system under most
+modern Linux distributions, and using systemd is strongly encouraged for production Galaxy deployments.
+
+Gravity manages `systemd service unit files`_ corresponding to all of the Galaxy services that it is aware of, much like
+how it manages supervisor program config files in supervisor mode. If you run ``galaxyctl update`` as a non-root user,
+the unit files will be installed in ``~/.config/systemd/user`` and run via `systemd user mode`_. This can be useful for
+testing and development, but in production it is recommended to run Gravity as root, so that it installs the service
+units in ``/etc/systemd/system`` and are managed by the privileged systemd instance. Even when Gravity is run as root,
+Galaxy itself still runs as a non-root user, specified by the ``galaxy_user`` option in the Gravity configuration.
+
+It is also recommended, when running as root, that you install Gravity independent of Galaxy, rather than use the copy
+installed in Galaxy's virtualenv:
+
+.. code:: console
+
+    # python3 -m venv /opt/gravity
+    # /opt/gravity/bin/pip install gravity
+
+.. caution::
+
+    Because systemd unit file names have semantic meaning (the filename is the service's name) and systemd does not have
+    a facility for isolating unit files controlled by an application, Gravity considers all unit files in the unit dir
+    (``/etc/systemd/system``) that are named like ``galaxy-*`` to be controlled by Gravity. **If you have existing unit
+    files that are named as such, Gravity will overwrite or remove them.**
+
+In systemd mode, and especially when run as root, some Gravity options are required:
+
+.. code:: yaml
+
+    gravity:
+      process_manager: systemd
+
+      # required if running as root
+      galaxy_user: GALAXY-USERNAME
+      # optional, defaults to primary group of the user set above
+      galaxy_group: GALAXY-GROUPNAME
+
+      # required
+      virtualenv: /srv/galaxy/venv
+      # probably necessary if your galaxy.yml is not in galaxy_root/config
+      galaxy_root: /srv/galaxy/server
+
+See the :ref:`Configuration` section for more details on these options and others.
+
+The ``log_dir`` option is ignored when using systemd. Logs are instead captured by systemd's logging facility,
+``journald``.
+
+You can use ``galaxyctl`` to manage Galaxy process starts/stops/restarts/etc. and follow the logs, just as you do under
+supervisor, but you can also use ``systemctl`` and ``journalctl`` directly to manage process states and inspect logs
+(respectively). Only ``galaxyctl update`` is necessary, in order to write and/or remove the appropriate systemd service
+units based on your configuration. For example:
+
+.. code:: console
+
+   # export GRAVITY_CONFIG_FILE=/srv/galaxy/config/galaxy.yml
+   # . /srv/galaxy/venv/bin/activate
+   (venv) # galaxyctl update
+   Adding service galaxy-gunicorn.service
+   Adding service galaxy-celery.service
+   Adding service galaxy-celery-beat.service
+
+After this point, operations can be performed with either ``galaxyctl`` or ``systemctl``. Some examples of equivalent
+commands:
+
+=================================== ==================================================================
+ Gravity                             systemd
+=================================== ==================================================================
+``galaxy``                          ``systemctl start galaxy.target && journalctl -f -u 'galaxy-*'``
+``galaxyctl start``                 ``systemctl start galaxy.target``
+``galaxyctl start SERVICE ...``     ``systemctl start galaxy-SERVICE.service galaxy-...``
+``galaxyctl restart``               ``systemctl restart galaxy.target``
+``galaxyctl restart SERVICE ...``   ``systemctl restart galaxy-SERVICE.service galaxy-...``
+``galaxyctl graceful``              ``systemctl reload-or-restart galaxy.target``
+``galaxyctl graceful SERVICE ...``  ``systemctl reload-or-restart galaxy-SERVICE.service galaxy-...``
+``galaxyctl stop``                  ``systemctl start galaxy.target``
+``galayxctl follow``                ``journalctl -f -u 'galaxy-*'``
+=================================== ==================================================================
+
+.. _supervisor: http://supervisord.org/
+.. _systemd: https://www.freedesktop.org/wiki/Software/systemd/
+.. _systemd service unit files: https://www.freedesktop.org/software/systemd/man/systemd.unit.html
+.. _systemd user mode: https://www.freedesktop.org/software/systemd/man/[email protected]

docs/conf.py

@@ -8,6 +8,8 @@
 import os
 import sys
 
+import sphinx_rtd_theme
+
 sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
 
 from gravity import __version__  # noqa: E402
@@ -31,5 +33,5 @@
 # -- Options for HTML output -------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
 
-html_theme = 'default'
-html_static_path = ['_static']
+html_theme = "sphinx_rtd_theme"
+html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]

docs/conventions.rst

@@ -0,0 +1,11 @@
+Documentation Conventions
+=========================
+
+Examples in this documentation assume a Galaxy layout like the one used in the `Galaxy Installation with Ansible`_
+tutorial::
+
+    /srv/galaxy/server  # Galaxy code
+    /srv/galaxy/config  # config files
+    /srv/galaxy/venv    # virtualenv
+
+.. _Galaxy Installation with Ansible: https://training.galaxyproject.org/training-material/topics/admin/tutorials/ansible-galaxy/tutorial.html

docs/history.rst

@@ -0,0 +1 @@
+.. include:: ../HISTORY.rst

docs/index.rst

@@ -6,8 +6,13 @@ Gravity
    :maxdepth: 2
    :caption: Contents:
 
-.. include:: ../README.rst
-.. include:: ../HISTORY.rst
+   readme
+   conventions
+   installation
+   basic_usage
+   advanced_usage
+   subcommands
+   history
 
 
 Indices and tables