Skip to content

Commit 008a511

Browse files
sei-dupdykesei-dupdyke
authored
Merge pull request #381 from cmu-sei/v8
updated docs from Austin W.
2 parents 49cd54d + a1e4f9a commit 008a511

File tree

3 files changed

+158
-64
lines changed

3 files changed

+158
-64
lines changed

‎docs/core/api.md

Lines changed: 60 additions & 64 deletions
Original file line numberDiff line numberDiff line change
@@ -1,107 +1,103 @@
1-
# GHOSTS Core API Overview
1+
# Setting Up the GHOSTS API
22

33
???+ info "GHOSTS Source Code"
44
The [GHOSTS Source Code Repository](https://github.com/cmu-sei/GHOSTS) is hosted on GitHub.
55

6+
*Updated on July 24, 2024*
7+
68
The GHOSTS API enables the control and orchestration of non-player characters (NPCs) within a deployment. It supports logging, reporting, and managing individual, groups of, or entire deployments of client installs.
79

8-
## Installation
10+
The GHOSTS API consists of three components: the API itself for configuring and managing characters and machines, a Postgres database for managing all the pieces, and Grafana for seeing GHOSTS activities in one convenient dashboard. Each of these three components runs in its own docker container.
911

10-
1. Install 🐳 [Docker](https://docs.docker.com/install/) :material-open-in-new:
11-
2. Install [Docker Compose](https://docs.docker.com/compose/install/) :material-open-in-new:
12-
3. Run the following commands - we'll use [this docker-compose.yml file](https://raw.githubusercontent.com/cmu-sei/GHOSTS/master/src/Ghosts.Api/docker-compose.yml)
12+
Steps to set up the GHOSTS API:
1313

14-
```cmd
15-
mkdir ghosts
16-
cd ghosts
17-
curl https://raw.githubusercontent.com/cmu-sei/GHOSTS/master/src/Ghosts.Api/docker-compose.yml -o docker-compose.yml
18-
docker-compose up -d
19-
```
14+
1. Choose where to host the API
15+
2. Install Docker and Docker Compose
16+
3. Build the GHOSTS containers
17+
4. Test the API
2018

21-
The required containers will be downloaded and configured automatically.
19+
## Step 1 — Choose Where to Host the API
2220

23-
Once the last command completes, if you open [http://localhost:5000/api/home](http://localhost:5000/api/home) in your browser, you should see the initial API page outlining the version of the install, and a few test machine entries. If this page renders, your API is up, running, and available.
21+
Choose the machine you'll be using to host the GHOSTS API. If you're just playing around with the GHOSTS, your local machine is fine. If you're configuring an exercise or simulation that other people might care about, consider creating a dedidicate host (server, image, or virtual server) for the API, or even using a container service such as AWS ECS.
2422

25-
You will still need to set up Grafana. Beware that you must often `chown` the host location of the container as listed in the docker-compose file or the container will just continually restart in error due to insufficient permissions.
23+
## Step 2 — Installing Docker
2624

27-
## Configuring the API
25+
You'll need to install Docker and Docker Compose on your API host.
2826

29-
The API generally has good defaults to get you up and running quickly, but there are some considerations in the `appconfig.json` file:
27+
First, **install 🐳 [Docker](https://docs.docker.com/install/)**.
3028

31-
```json
32-
"ClientSettings": {
33-
"OfflineAfterMinutes": 30, ...
34-
"MatchMachinesBy": null,
35-
```
29+
Next, **install [Docker Compose](https://docs.docker.com/compose/install/)** which will assist with starting up multiple containers at the same time.
3630

37-
Can be fqdn|host|resolvedhost|null - null tells the API to match incoming requests with machine records by the machine name. For installations where multiple domains are reporting into the same API, you probably want to use FQDN in order to avoid machines being duplicated.
31+
Before continuing, test that you have the command for Docker Compose available.
3832

33+
Open your system's CMD (Linux), Terminal (Mac), or PowerShell (Windows).
3934

40-
```json
41-
"QueueSyncDelayInSeconds": 10,
42-
"NotificationsQueueSyncDelayInSeconds": 10,
35+
```
36+
$ docker-compose --version
4337
```
4438

45-
This is how often the synch job runs. Incoming machine requests are not real-time in order to best bundle like records together.
39+
## Step 3 — Installing the GHOSTS API
4640

47-
## Configuring Grafana
41+
Once you have confirmed that Docker and Docker Compose are installed, you can build the containers required for the GHOSTS API.
4842

49-
- Grafana will be running (if containerized) on port 3000, and we can access it via the same URL we use for the API.
50-
- The default login is admin/admin.
51-
- The first step is to set up a datasource named "ghosts" to the ghosts Postgres database.
52-
- Now import your choice of the [grafana json files](https://github.com/cmu-sei/GHOSTS/tree/master/configuration/grafana) in this repository. It creates the default GHOSTS dashboard.
43+
Create a directory where you want to store the build files and containers.
5344

54-
## Webhooks
45+
```
46+
$ mkdir ghosts-project
47+
$ cd ghosts-project
48+
```
5549

56-
The GHOSTS API provides webhook callbacks based on the configuration on the endpoint: `/api/webhooks`. The payload for creating a webhook is in the format:
50+
Download the docker compose file for GHOSTS.
5751

58-
```json
59-
{
60-
"status": 0,
61-
"description": "some description",
62-
"postbackUrl": "http://localhost/endpoint:port",
63-
"postbackMethod": 0, (0 == get, 1 == post)
64-
"postbackFormat": "see below"
65-
}
52+
```
53+
$ curl https://raw.githubusercontent.com/cmu-sei/GHOSTS/master/src/Ghosts.Api/docker-compose.yml -o docker-compose.yml
6654
```
6755

68-
Payloads can be any format — here is a sample:
56+
Build all of the containers at once using docker-compose.
6957

70-
```json
71-
{
72-
'machine':'[machinename]',
73-
'created':'[datetime.utcnow]',
74-
'type':'[messagetype]',
75-
'payload':'[messagepayload]'
76-
}
58+
```
59+
$ docker-compose up -d
7760
```
7861

79-
On send, the payload will be converted into the correct JSON format:
62+
Check for the running containers.
8063

81-
```json
82-
{
83-
"machine":"some_guid",
84-
"created":"some_datetime",
85-
"type":"some_message",
86-
"payload":"some_payload"
87-
}
8864
```
65+
$ docker ps -a
66+
```
67+
68+
If everything succeeds you should see the three new containers for the API, Grafana, and Postgres.
8969

90-
If the postback method is POST, the payload will be sent as the message body. If the postback method is GET, the payload will be sent as part of the querystring value ?message=`payload`.
70+
![Running Containers](../images/installing-the-api-running-containers.png)
9171

92-
The following events are reported via webhooks:
72+
## Step 4 — Testing the API
9373

94-
1. Timeline delivered (with the timeline that was delivered as payload) to a machine via API (original API posting of timeline only holds timeline in wait - the client still must check-in in order for that timeline to be delivered)
95-
2. Machine requested updates ("checked in") from API
96-
3. Machine posted results to API
74+
By default, the API is hosted on port 5000. You should be able to reach the API from [http://localhost:5000](http://localhost:5000). If you open this page in your browser, you should see the initial API page outlining the version of the install, and a few test machine entries. If this page renders, your API is up, running, and available.
75+
76+
![Success!](../images/installing-the-api-success.png)
9777

9878
## Troubleshooting
9979

100-
> Is the API up and running?
80+
### Problem: The API home page has an error
81+
82+
![API Home Page Error](../images/installing-the-api-error.png)
83+
84+
Answer: Make sure the docker container for Postgres is running using Docker Desktop or the command `docker ps -a`
85+
86+
![Running Containers](../images/installing-the-api-running-containers.png)
87+
88+
You can check the logs with the command `docker logs ghosts-postgres` to look for container errors.
89+
90+
### Problem: The social graph link has an error
91+
92+
![API Social Graph Page Error](../images/installing-the-api-social-error.png)
93+
94+
Answer: You haven't created a social network yet, this is normal.
95+
96+
### Problem: Is the API up and running?
10197

10298
- Go to `/api/home` in the browser, it should return the current API version and the number of machines and groups under management. If it says relationship not found, restart the API application and it should create the database automatically.
10399
- Run `docker ps --all` and see that all containers are running normally. If one or more is not running, look at the logs for that machine via `docker logs [machine name]`.
104100

105101
> The ClientId, ClientResults, and other Client* endpoints are failing.
106102
107-
The Client* endpoints are for the Clients to use only. There are specific header values set by the client in the request that is used to authenticate the request. If you are not using the client, you will not have these headers set, and these endpoints will fail.
103+
The Client* endpoints are for the Clients to use only. There are specific header values set by the client in the request that is used to authenticate the request. If you are not using the client, you will not have these headers set, and these endpoints will fail.

‎docs/core/grafana.md

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
# Configuring Grafana
2+
3+
*Updated on July 25, 2024*
4+
5+
[Grafana](https://grafana.com/) for GHOSTS allows the simulation administrator to visualize all activities carried out by the NPCs across the simulation in dashboard with pretty colors and charts.
6+
7+
## Prerequisites
8+
9+
- Your GHOSTS API should be installed and running (see [Setting Up the GHOSTS API](installing-the-api.md))
10+
11+
## Step 1 — Container Set Up
12+
13+
The Grafana docker container will be installed during the process of [Setting Up the GHOSTS API](installing-the-api.md). Depending on your system and account, the Grafana container may or may not have the permissions it needs to run.
14+
15+
You can check its status with the docker command
16+
17+
```
18+
$ docker ps -a
19+
```
20+
21+
If the container is continuously restarting, Grafana does not have the permissions it needs.
22+
23+
![Grafana Restarting](../images/configuring-grafana-restarting.png)
24+
25+
You can also check the docker logs:
26+
27+
```
28+
$ docker logs ghosts-grafana
29+
```
30+
31+
![Grafana Permission Denied](../images/configuring-grafana-permission-denied.png)
32+
33+
If you don't see this issue, you can continue to Step 2. If you do see this issue you will need to grant permissions on the `_g` directory of the ghosts-api folder (which stores the Grafana data for the GHOSTS API).
34+
35+
```
36+
$ cd ghosts-api
37+
$ chmod 777 _g
38+
```
39+
40+
Ensure the container is running with `docker ps`.
41+
42+
## Step 2 — Configuring Grafana
43+
44+
Once the container is running you can access its front end by default at [localhost:3000](http://localhost:3000)
45+
46+
![Grafana Front end](../images/configuring-grafana-front-end.png)
47+
48+
The default login is:
49+
50+
| Username | Password |
51+
| -------- | -------- |
52+
| admin | admin |
53+
54+
Continue through the setup prompts.
55+
56+
### Step 3 — Setting the Datasource
57+
58+
Now you need to tell Grafana where it will be getting its data.
59+
60+
1. Open the "Connections" from the left menu.
61+
2. Click on "Add new data source" in the top right corner
62+
3. Search for "Postgres" and choose the PostgreSQL option
63+
4. Name the datasource "ghosts" and leave it as the default
64+
5. Under the "Connection" section of the config, set
65+
- host url to "ghosts-postgres:5432"
66+
- database name to "ghosts"
67+
6. Under the "Authentication" section of the config, set
68+
- username to "ghosts"
69+
- password to "scotty@1"
70+
- TLS/SSL Mode to "disable"
71+
7. Leave everything else at its default and click the "Save and test" button at the bottom of the page
72+
73+
74+
### Step 4 — Choosing a Dashboard
75+
76+
Grafana dashboards are very flexible and can be configured to show any statistics that are important to your simulation or experiment.
77+
78+
GHOSTS comes with some premade dashboards to get you started. You can download those here: https://github.com/cmu-sei/GHOSTS/tree/master/configuration/grafana/dashboards
79+
80+
- GHOSTS-5-default Grafana dashboard — shows status across all machines
81+
- GHOSTS-5-group-default Grafana dashboard — shows status with machines grouped by enclave
82+
83+
#### Loading an existing dashboard
84+
85+
Navigate to "Dashboards" in the left menu. There will be a blue "New" button in the top right corner.
86+
87+
![Empty Dashboard](../images/configuring-grafana-empty-dashboard.png)
88+
89+
Click "New". Then, "import".
90+
91+
You can either upload one of the dashboard json files from the GHOSTS repository or simply copy and paste the json into the "import via dashboard json model" panel.
92+
93+
Choose the ghosts datasource you added earlier from the drop down menu and then click "import".
94+
95+
![Ghosts In the Dashboard](../images/configuring-grafana-dashboard.png)
96+
97+
You are now set up with Grafana!

‎mkdocs.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,7 @@ nav:
9494
- core/api.md
9595
- Capabilities:
9696
- Managing Timelines: core/api/timelines.md
97+
- Grafana: core/grafana.md
9798
- Animations:
9899
- animator/index.md
99100
- Animation Jobs: animator/jobs.md

0 commit comments

Comments
 (0)