OperationsGateway API

This API is built using FastAPI to work with MongoDB and ECHO S3. It allows CRUD access to the data stored as part of the OperationsGateway project for the EPAC facility.

A separate repository exists to manage the deployment on pre-production and production machines using the Ansible framework.

The following instructions will detail how to get a development instance up and running. They can run on a command line and are tailored for a new rocky-9-nogui virtual machine with SSH access.

Prerequisites

1. Local development on Windows or Mac can be challenging because they lack the Linux libraries required by the API. Since the API is designed to run in a Python 3.11 environment on a Rocky 9 Linux machine, it’s best to use a Rocky 9 development VM to closely mirror the production setup.
2. You'll need a developer bucket in echo to use the test script to prefill echo, and a local database, with data.
3. One of the dependencies used in this API (epac-data-sim) is a private repository, so the appropriate permissions and SSH keys need to be set up. Guidance for setting up SSH keys for the Rocky 9 VM can be found here.

Environment Setup

Python

See commands below to install the development tools for specific operating systems. Others will need the same libraries, but exact names may differ.

Rocky 9

sudo dnf install "@Development Tools" python3.11 python3.11-pip python3.11-setuptools python3.11-devel openldap-devel swig gcc gcc-c++ openssl-devel cmake3 libuv libuuid-devel git

Debian 12

sudo agt-get update
sudo apt-get install build-essential curl python3.11 python3.11-venv python3.11-dev python3-pip libldap2-dev libsasl2-dev cmake uuid-dev git

Clone the codebase

git clone https://github.com/ral-facilities/operationsgateway-api.git

Poetry

Poetry is used to manage the dependencies of this API.

# Install poetry
curl -sSL https://install.python-poetry.org | python3 -
# Change the directory to where the poetry .toml file is
cd operationsgateway-api
# Set poetry to use python 3.11
poetry env use python3.11
# Install the dependencies
poetry install
# If you don't need the simulated data, or don't have access to the epac repo...
poetry install --without simulated-data

MongoDB:

Create the following file:

sudo vi /etc/yum.repos.d/mongodb-org-7.0.repo

Add:

[mongodb-org-7.0]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/9/mongodb-org/7.0/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://pgp.mongodb.com/server-7.0.asc

Then:

# Install MongoDB
sudo yum install mongodb-org
# Start it
sudo systemctl start mongod
# Enable it to start on boot
sudo systemctl enable mongod

For instructions on installation on specific operating systems:

https://www.mongodb.com/docs/manual/administration/install-on-linux/

MongoDB Indexes:

The following Indexes are uses and need to be set up on local, dev & prod databases.

Using mongosh, select the database: use opsgateway

Then run the following command:

# Multiple users can have the same session name, but a user can't have two sessions with the same name.
db.sessions.createIndex(
  { username: 1, name: 1 },
  { unique: true }
);

Authentication

The authentication system uses JSON Web Tokens (JWTs), which require a private and public key pair to encrypt and decrypt the tokens.

The keys can be generated by navigating to a directory and typing the command:

# When prompted for the location path, enter `id_rsa` to create the keys in the current directory rather than in your home directory.
# Press enter twice when prompted for the password so as not to set one.
ssh-keygen -b 2048 -t rsa

Then, edit the private_key_path and public_key_path settings in the auth section of the config.yml file to reflect the location where these keys have been created.

API Configuration

In operationsgateway_api/, there are several example configuration files which need to be reviewed and renamed, removing the .example part of the filename.

• config.yml.example
• logging.ini.example
• maintenance.json.example
• scheduled_maintenance.json.example

Backup

If using the tape backup feature (enabled by the backup section in config.yml) then authentication for XRootD will need to be configured. XRootD uses environment variables to define the credentials used to authenticate to the server. These can either be explicitly defined in the session as environment variables or in config.yml, which case this will be set as an environment variable on start up. Note that files containing keys need to be only readable/writable by the user, so it may be necessary to manually configure this:

chmod 600 /path/to/.keytab
export XrdSecPROTOCOL=sss
export XrdSecSSSKT=/path/to/.keytab

Test Data

A script has been created to set up the API with some test data and test users. This script also comes with a configuration file that needs to be reviewed and renamed at util/realistic_data/config.yml.example.

Once the appropriate configuration has been entered, run the script using Poetry:

poetry run python util/realistic_data/ingest_echo_data.py

More details about this script can be found in docs/developer/test_data.md

Starting the API

poetry run python operationsgateway_api/src/main.py

Assuming the default configuration, the API will exist on 127.0.0.1:8000.

The port forwarding service on most IDEs will allow you to access the API on the remote VM through your local web browser. Then, you can visit /docs in a browser, which will give an OpenAPI interface detailing each of the endpoints.

Alternatively, you can use the Postman requests in this repo. This gives a more user-friendly way of interacting with the API. Certain environments and scripts have been set up in Postman to help with using the API. For example, before any call is made to the API, the /login call can be used to get and store a JWT, which will be used automatically in the auth header of following calls to the API.

Testing

Like the DataGateway API, this repository contains a Nox file (noxfile.py), which exists at the root level of this repository.

To install Nox, use the following command:

python3.11 -m pip install --user --upgrade nox

To run a specific Nox session, use the following:

nox -s [SESSION NAME]

The following Nox sessions have been created:

• black - This uses Black to format Python code in a pre-defined style.
• lint - This uses flake8 with a number of additional plugins to lint the code to keep it Pythonic.
• safety - This uses safety to check the dependencies (pulled directly from Poetry) for any known vulnerabilities.
• tests - this uses pytest to execute the automated tests in test/.