Skip to content

Commit ad14117

Browse files
natefoonatefoo
committed
Properly organize documentation instead of sticking it all in the README
1 parent a1fce5b commit ad14117

File tree

11 files changed

+982
-857
lines changed

11 files changed

+982
-857
lines changed

‎README.rst

Lines changed: 37 additions & 853 deletions
Large diffs are not rendered by default.

‎docs/advanced_usage.rst

Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
Advanced Usage
2+
==============
3+
4+
Zero-Downtime Restarts
5+
----------------------
6+
7+
Prior to Gravity 1.0, the preferred solution for performing zero-downtime restarts was `unicornherder`_. However, due to
8+
limitations in the unicornherder software, it does not always successfully perform zero-downtime restarts. Because of
9+
this, Gravity is now able to perform rolling restarts of gunicorn services if more than one gunicorn is configured.
10+
11+
To run multiple gunicorn processes, configure the ``gunicorn`` section of the Gravity configuration as a *list*. Each
12+
item in the list is a gunicorn configuration, and can have all of the same parameters as a single gunicorn
13+
configuration:
14+
15+
.. code:: yaml
16+
17+
gravity:
18+
gunicorn:
19+
- bind: unix:/srv/galaxy/var/gunicorn0.sock
20+
workers: 4
21+
- bind: unix:/srv/galaxy/var/gunicorn1.sock
22+
workers: 4
23+
24+
.. caution::
25+
26+
This will start multiple Galaxy servers with the same ``server_name``. If you have not configured separate Galaxy
27+
processes to act as job handlers, your gunicorn processes will handle them, resulting in job errors due to handling
28+
the same job multiple times. See the Gravity and Galaxy documentation on configuring handlers.
29+
30+
Your proxy server can balance load between the two gunicorns. For example, with nginx:
31+
32+
.. code:: nginx
33+
34+
upstream galaxy {
35+
server unix:/srv/galaxy/var/gunicorn0.sock;
36+
server unix:/srv/galaxy/var/gunicorn1.sock;
37+
}
38+
39+
http {
40+
location / {
41+
proxy_pass http://galaxy;
42+
}
43+
}
44+
45+
By default, Gravity will wait 300 seconds for the gunicorn server to respond to web requests after initiating the
46+
restart. To change this timeout this, set the ``restart_timeout`` option on each configured ``gunicorn`` instance.
47+
48+
Service Instances
49+
-----------------
50+
51+
In the case of multiple gunicorn instances as described in :ref:`Zero-Downtime Restarts` and multiple dynamic handlers
52+
as described in :ref:`Galaxy Job Handlers`, Gravity will create multiple *service instances* of each service. This
53+
allows multiple processes to be run from a single service definition.
54+
55+
In supervisor, this means that the service names as presented by supervisor are appended with ``:INSTANCE_NUMBER``,
56+
e.g.:
57+
58+
.. code:: console
59+
60+
$ galaxyctl status
61+
celery RUNNING pid 121363, uptime 0:02:33
62+
celery-beat RUNNING pid 121364, uptime 0:02:33
63+
gunicorn:0 RUNNING pid 121365, uptime 0:02:33
64+
gunicorn:1 RUNNING pid 121366, uptime 0:02:33
65+
66+
However, ``galaxyctl`` commands that take a service name still use the base service name, e.g.:
67+
68+
.. code:: console
69+
70+
$ galaxyctl stop gunicorn
71+
gunicorn:0: stopped
72+
gunicorn:1: stopped
73+
Not all processes stopped, supervisord not shut down (hint: see `galaxyctl status`)
74+
75+
In systemd, the service names as presented by systemd are appended with ``@INSTANCE_NUMBER``,
76+
e.g.:
77+
78+
.. code:: console
79+
80+
$ galaxyctl status
81+
UNIT LOAD ACTIVE SUB DESCRIPTION
82+
galaxy-celery-beat.service loaded active running Galaxy celery-beat
83+
galaxy-celery.service loaded active running Galaxy celery
84+
[email protected] loaded active running Galaxy gunicorn (process 0)
85+
[email protected] loaded active running Galaxy gunicorn (process 1)
86+
galaxy.target loaded active active Galaxy
87+
88+
As with supervisor, ``galaxyctl`` commands that take a service name still use the base service name.
89+
90+
If you prefer not to work with service instances and want Galaxy to write a service configuration file for each instance
91+
of each service, you can do so by setting ``use_service_instances`` in the Gravity configuration to ``false``.
92+
93+
Managing Multiple Galaxies
94+
--------------------------
95+
96+
Gravity can manage multiple instances of Galaxy simultaneously. This is useful especially in the case where you have
97+
multiple production Galaxy instances on a single server and are managing them with Gravity installed outside of a Galaxy
98+
virtualenv, as root. There are multiple ways to achieve this:
99+
100+
1. Pass multiple ``--config-file`` options to ``galaxyctl``, or set a list of colon-separated config paths in
101+
``$GRAVITY_CONFIG_FILE``:
102+
103+
.. code:: console
104+
105+
$ galaxyctl --config-file /srv/galaxy/test/config/galaxy.yml \
106+
--config-file /srv/galaxy/main/config/galaxy.yml list --version
107+
TYPE INSTANCE NAME VERSION CONFIG PATH
108+
galaxy test 22.05 /srv/galaxy/test/config/galaxy.yml
109+
galaxy main 22.09.dev0 /srv/galaxy/main/config/galaxy.yml
110+
$ export GRAVITY_CONFIG_FILE='/srv/galaxy/test/config/galaxy.yml:/srv/galaxy/main/config/galaxy.yml'
111+
$ galaxyctl list --version
112+
TYPE INSTANCE NAME VERSION CONFIG PATH
113+
galaxy test 22.05 /srv/galaxy/test/config/galaxy.yml
114+
galaxy main 22.09.dev0 /srv/galaxy/main/config/galaxy.yml
115+
116+
2. If running as root, any config files located in ``/etc/galaxy/gravity.d`` will automatically be loaded.
117+
118+
3. Specify multiple Gravity configurations in a single config file, as a list. In this case, the Galaxy and Gravity
119+
configurations must be in separate files as described in :ref:`Splitting Gravity and Galaxy Configurations`:
120+
121+
.. code:: yaml
122+
123+
gravity:
124+
- instance_name: test
125+
process_manager: systemd
126+
galaxy_config_file: /srv/galaxy/test/config/galaxy.yml
127+
galaxy_root: /srv/galaxy/test/server
128+
virtualenv: /srv/galaxy/test/venv
129+
galaxy_user: gxtest
130+
gunicorn:
131+
bind: unix:/srv/galaxy/test/var/gunicorn.sock
132+
handlers:
133+
handler:
134+
pools:
135+
- job-handlers
136+
- workflow-schedulers
137+
138+
- instance_name: main
139+
process_manager: systemd
140+
galaxy_config_file: /srv/galaxy/main/config/galaxy.yml
141+
galaxy_root: /srv/galaxy/main/server
142+
virtualenv: /srv/galaxy/main/venv
143+
galaxy_user: gxmain
144+
gunicorn:
145+
bind: unix:/srv/galaxy/main/var/gunicorn.sock
146+
workers: 8
147+
handlers:
148+
handler:
149+
processes: 4
150+
pools:
151+
- job-handlers
152+
- workflow-schedulers
153+
154+
In all cases, when using multiple Gravity instances, each Galaxy instance managed by Gravity must have a unique
155+
**instance name**. When working with a single instance, the default name ``_default_`` is used automatically and mostly
156+
hidden from you. When working with multiple instances, set the ``instance_name`` option in each instance's Gravity
157+
config to a unique name.
158+
159+
Although it is strongly encouraged to use systemd for running multiple instances, it is possible to use supervisor.
160+
Please see the :ref:`Gravity State` section for important details on how and where Gravity stores the supervisor
161+
configuration and log files.
162+
163+
.. _unicornherder: https://github.com/alphagov/unicornherder

‎docs/basic_usage.rst

Lines changed: 176 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,176 @@
1+
Basic Usage
2+
===========
3+
4+
A basic example of starting and running a simple Galaxy server from a source clone in the foreground is provided in the
5+
ref:`Quick Start` guide. This section covers more typical usage for production Galaxy servers.
6+
7+
Managing a Single Galaxy
8+
------------------------
9+
10+
If you have not installed Gravity separate from the Galaxy virtualenv, simply activate Galaxy's virtualenv, which will
11+
put Gravity's ``galaxyctl`` and ``galaxy`` commands on your ``$PATH``:
12+
13+
.. code:: console
14+
15+
$ . /srv/galaxy/venv/bin/activate
16+
$ galaxyctl --help
17+
Usage: galaxyctl [OPTIONS] COMMAND [ARGS]...
18+
19+
Manage Galaxy server configurations and processes.
20+
21+
Options:
22+
-d, --debug Enables debug mode.
23+
-c, --config-file FILE Gravity (or Galaxy) config file to operate on. Can
24+
also be set with $GRAVITY_CONFIG_FILE or
25+
$GALAXY_CONFIG_FILE
26+
--state-dir DIRECTORY Where process management configs and state will be
27+
stored.
28+
-h, --help Show this message and exit.
29+
30+
Commands:
31+
configs List registered config files.
32+
deregister Deregister config file(s).
33+
exec Run a single Galaxy service in the foreground, with logging...
34+
follow Follow log files of configured instances.
35+
graceful Gracefully reload configured services.
36+
instances List all known instances.
37+
pm Invoke process manager (supervisorctl, systemctl) directly.
38+
register Register config file(s).
39+
rename Update path of registered config file.
40+
restart Restart configured services.
41+
show Show details of registered config.
42+
shutdown Shut down process manager.
43+
start Start configured services.
44+
status Display server status.
45+
stop Stop configured services.
46+
update Update process manager from config changes.
47+
48+
If you run ``galaxy`` or ``galaxyctl`` from the root of a Galaxy source checkout and do not specify the config file
49+
option, ``config/galaxy.yml`` or ``config/galaxy.yml.sample`` will be automatically used. This is handy for working with
50+
local clones of Galaxy for testing or development. You can skip Galaxy's lengthy and repetitive ``run.sh`` configuration
51+
steps when starting and stopping Galaxy in between code updates (you should still run ``run.sh`` after performing a
52+
``git pull`` to make sure your dependencies are up to date).
53+
54+
Gravity can either run Galaxy via the `supervisor`_ process manager (the default) or `systemd`_. For production servers,
55+
**it is recommended that you run Gravity as root in systemd mode**. See the :ref:`Managing a Production Galaxy` section
56+
for details.
57+
58+
As shown in the Quick Start, the ``galaxy`` command will run a Galaxy server in the foreground. The ``galaxy`` command
59+
is actually a shortcut for two separate steps: 1. read the provided ``galaxy.yml`` and write out the corresponding
60+
process manager configurations, and 2. start and run Galaxy in the foreground using the process manager (`supervisor`_).
61+
You can perform these steps separately (and in this example, start Galaxy as a backgrounded daemon instead of in the
62+
foreground):
63+
64+
.. code:: console
65+
66+
$ galaxyctl update
67+
Adding service gunicorn
68+
Adding service celery
69+
Adding service celery-beat
70+
$ galaxyctl start
71+
celery STARTING
72+
celery-beat STARTING
73+
gunicorn STARTING
74+
Log files are in /srv/galaxy/var/gravity/log
75+
76+
When running as a daemon, the ``stop`` subcommand stops your Galaxy server:
77+
78+
.. code:: console
79+
80+
$ galaxyctl stop
81+
celery-beat: stopped
82+
gunicorn: stopped
83+
celery: stopped
84+
All processes stopped, supervisord will exit
85+
Shut down
86+
87+
Most Gravity subcommands (such as ``stop``, ``start``, ``restart``, ...) are straightforward, but a few subcommands are
88+
worth pointing out: :ref:`update`, :ref:`graceful`, and :ref:`exec`. All subcommands are documented in the
89+
:ref:`Subcommands` section and their respective ``--help`` output.
90+
91+
Managing a Production Galaxy
92+
----------------------------
93+
94+
By default, Gravity runs Galaxy processes under `supervisor`_, but setting the ``process_manager`` option to ``systemd``
95+
in Gravity's configuration will cause it to run under `systemd`_ instead. systemd is the default init system under most
96+
modern Linux distributions, and using systemd is strongly encouraged for production Galaxy deployments.
97+
98+
Gravity manages `systemd service unit files`_ corresponding to all of the Galaxy services that it is aware of, much like
99+
how it manages supervisor program config files in supervisor mode. If you run ``galaxyctl update`` as a non-root user,
100+
the unit files will be installed in ``~/.config/systemd/user`` and run via `systemd user mode`_. This can be useful for
101+
testing and development, but in production it is recommended to run Gravity as root, so that it installs the service
102+
units in ``/etc/systemd/system`` and are managed by the privileged systemd instance. Even when Gravity is run as root,
103+
Galaxy itself still runs as a non-root user, specified by the ``galaxy_user`` option in the Gravity configuration.
104+
105+
It is also recommended, when running as root, that you install Gravity independent of Galaxy, rather than use the copy
106+
installed in Galaxy's virtualenv:
107+
108+
.. code:: console
109+
110+
# python3 -m venv /opt/gravity
111+
# /opt/gravity/bin/pip install gravity
112+
113+
.. caution::
114+
115+
Because systemd unit file names have semantic meaning (the filename is the service's name) and systemd does not have
116+
a facility for isolating unit files controlled by an application, Gravity considers all unit files in the unit dir
117+
(``/etc/systemd/system``) that are named like ``galaxy-*`` to be controlled by Gravity. **If you have existing unit
118+
files that are named as such, Gravity will overwrite or remove them.**
119+
120+
In systemd mode, and especially when run as root, some Gravity options are required:
121+
122+
.. code:: yaml
123+
124+
gravity:
125+
process_manager: systemd
126+
127+
# required if running as root
128+
galaxy_user: GALAXY-USERNAME
129+
# optional, defaults to primary group of the user set above
130+
galaxy_group: GALAXY-GROUPNAME
131+
132+
# required
133+
virtualenv: /srv/galaxy/venv
134+
# probably necessary if your galaxy.yml is not in galaxy_root/config
135+
galaxy_root: /srv/galaxy/server
136+
137+
See the :ref:`Configuration` section for more details on these options and others.
138+
139+
The ``log_dir`` option is ignored when using systemd. Logs are instead captured by systemd's logging facility,
140+
``journald``.
141+
142+
You can use ``galaxyctl`` to manage Galaxy process starts/stops/restarts/etc. and follow the logs, just as you do under
143+
supervisor, but you can also use ``systemctl`` and ``journalctl`` directly to manage process states and inspect logs
144+
(respectively). Only ``galaxyctl update`` is necessary, in order to write and/or remove the appropriate systemd service
145+
units based on your configuration. For example:
146+
147+
.. code:: console
148+
149+
# export GRAVITY_CONFIG_FILE=/srv/galaxy/config/galaxy.yml
150+
# . /srv/galaxy/venv/bin/activate
151+
(venv) # galaxyctl update
152+
Adding service galaxy-gunicorn.service
153+
Adding service galaxy-celery.service
154+
Adding service galaxy-celery-beat.service
155+
156+
After this point, operations can be performed with either ``galaxyctl`` or ``systemctl``. Some examples of equivalent
157+
commands:
158+
159+
=================================== ==================================================================
160+
Gravity systemd
161+
=================================== ==================================================================
162+
``galaxy`` ``systemctl start galaxy.target && journalctl -f -u 'galaxy-*'``
163+
``galaxyctl start`` ``systemctl start galaxy.target``
164+
``galaxyctl start SERVICE ...`` ``systemctl start galaxy-SERVICE.service galaxy-...``
165+
``galaxyctl restart`` ``systemctl restart galaxy.target``
166+
``galaxyctl restart SERVICE ...`` ``systemctl restart galaxy-SERVICE.service galaxy-...``
167+
``galaxyctl graceful`` ``systemctl reload-or-restart galaxy.target``
168+
``galaxyctl graceful SERVICE ...`` ``systemctl reload-or-restart galaxy-SERVICE.service galaxy-...``
169+
``galaxyctl stop`` ``systemctl start galaxy.target``
170+
``galayxctl follow`` ``journalctl -f -u 'galaxy-*'``
171+
=================================== ==================================================================
172+
173+
.. _supervisor: http://supervisord.org/
174+
.. _systemd: https://www.freedesktop.org/wiki/Software/systemd/
175+
.. _systemd service unit files: https://www.freedesktop.org/software/systemd/man/systemd.unit.html
176+
.. _systemd user mode: https://www.freedesktop.org/software/systemd/man/[email protected]

‎docs/conf.py

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@
88
import os
99
import sys
1010

11+
import sphinx_rtd_theme
12+
1113
sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
1214

1315
from gravity import __version__ # noqa: E402
@@ -31,5 +33,5 @@
3133
# -- Options for HTML output -------------------------------------------------
3234
# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
3335

34-
html_theme = 'default'
35-
html_static_path = ['_static']
36+
html_theme = "sphinx_rtd_theme"
37+
html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]

‎docs/conventions.rst

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
Documentation Conventions
2+
=========================
3+
4+
Examples in this documentation assume a Galaxy layout like the one used in the `Galaxy Installation with Ansible`_
5+
tutorial::
6+
7+
/srv/galaxy/server # Galaxy code
8+
/srv/galaxy/config # config files
9+
/srv/galaxy/venv # virtualenv
10+
11+
.. _Galaxy Installation with Ansible: https://training.galaxyproject.org/training-material/topics/admin/tutorials/ansible-galaxy/tutorial.html

‎docs/gravity-logo.png

89.9 KB
Loading

‎docs/history.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
.. include:: ../HISTORY.rst

‎docs/index.rst

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,13 @@ Gravity
66
:maxdepth: 2
77
:caption: Contents:
88

9-
.. include:: ../README.rst
10-
.. include:: ../HISTORY.rst
9+
readme
10+
conventions
11+
installation
12+
basic_usage
13+
advanced_usage
14+
subcommands
15+
history
1116

1217

1318
Indices and tables

0 commit comments

Comments
 (0)