Skip to content

Commit 5c58136

Browse files
tlvutlvu
authored
Merge pull request #41 from bird-house/add-finch2-optional-component
 Add generic_bird optional component. Could be used as work-around for issue bird-house/finch#121 so we have a Finch dedicated for async request only. This component is generic and configurable enough it could even accommodate any WPS. It has its own Postgres instance so we can use it later to experiment with different version of Postgres. Test server: https://lvupavics-lvu.pagekite.me/twitcher/ows/proxy/finchasync, https://lvupavics-lvu.pagekite.me/canarie/node/service/status See the `README.md` update for more info. Note the docker image, service name and port is customizable (my test server do not use the default service name). It starts out as a Finch2 but now its rather a generic skeleton to plugin any birds we need. I wish I had this when I was testing out the new Thunderbird from PCIC. At some point in the past, we had the idea of 2 simultaneous FlyingPigeon because the new one is so different we wanted to keep the old one around for a short while. This PR could have helped. So do not view this PR as Finch 2 anymore, view this PR as a mechanism to quickly deploy another bird, any bird, without having to change code. This opens up many options, testing is just one of the possible usage.
2 parents 6036dbd + 817c6f4 commit 5c58136

14 files changed

+139
-3
lines changed

‎birdhouse/common.env

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,11 @@
33
# Jupyter single-user server image
44
export DOCKER_NOTEBOOK_IMAGE="pavics/workflow-tests:200312"
55

6+
export FINCH_IMAGE="birdhouse/finch:version-0.5.2"
7+
export GENERIC_BIRD_IMAGE="$FINCH_IMAGE"
8+
export GENERIC_BIRD_PORT="8010"
9+
export GENERIC_BIRD_NAME="generic_bird"
10+
611
# Folder on the host to persist Jupyter user data (noteboooks, HOME settings)
712
export JUPYTERHUB_USER_DATA_DIR="/data/jupyterhub_user_data"
813

‎birdhouse/docker-compose.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -151,7 +151,7 @@ services:
151151
restart: always
152152

153153
finch:
154-
image: birdhouse/finch:version-0.5.2
154+
image: ${FINCH_IMAGE}
155155
container_name: finch
156156
environment:
157157
HOSTNAME: $HOSTNAME

‎birdhouse/optional-components/README.md

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,3 +51,33 @@ server.
5151

5252
No Twitcher/Magpie access control, this Test Thredds is directly behind the
5353
Nginx proxy.
54+
55+
56+
## A generic bird WPS service
57+
58+
Can be used to quickly deploy any birds temporarily without changing code.
59+
Good to preview new birds or test alternative configuration of existing birds.
60+
61+
How to enable in `env.local` (a copy from
62+
[`env.local.example`](../env.local.example)):
63+
64+
* Add `./optional-components/generic_bird` to `EXTRA_CONF_DIRS`.
65+
66+
* Optionally set `GENERIC_BIRD_IMAGE`, `GENERIC_BIRD_PORT`, `GENERIC_BIRD_NAME` in `env.local`
67+
for further customizations. Default values are in [`common.env`](../common.env).
68+
69+
The WPS service will be available at `http://PAVICS_FQDN:GENERIC_BIRD_PORT/wps`
70+
or `https://PAVICS_FQDN_PUBLIC/TWITCHER_PROTECTED_PATH/GENERIC_BIRD_NAME` where
71+
`PAVICS_FQDN`, `PAVICS_FQDN_PUBLIC` and `TWITCHER_PROTECTED_PATH` are defined
72+
in your `env.local`.
73+
74+
Use same docker image as regular Finch by default but can be customized.
75+
76+
Use a separate Postgres DB for this optional component to be completely
77+
self-contained and to allow experimenting with different versions of Postgres
78+
DB.
79+
80+
Magpie will be automatically configured to give complete public anonymous
81+
access for this WPS service.
82+
83+
Canarie monitoring will also be automatically configured for this WPS service.

‎birdhouse/optional-components/generic_bird/.gitignore

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
wps.cfg
2+
generic_bird-magpie-permission.cfg
3+
generic_bird-magpie-provider.cfg
4+
generic_bird_canarie_api_monitoring.py

‎birdhouse/optional-components/generic_bird/docker-compose-extra.yml

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
version: '2.1'
2+
services:
3+
generic_bird:
4+
image: ${GENERIC_BIRD_IMAGE}
5+
container_name: generic_bird
6+
environment:
7+
PYWPS_CFG: /wps.cfg
8+
ports:
9+
- "${GENERIC_BIRD_PORT}:${GENERIC_BIRD_INTERNAL_PORT:-5000}"
10+
volumes:
11+
- ./optional-components/generic_bird/wps.cfg:/wps.cfg:ro
12+
- wps_outputs:/data/wpsoutputs
13+
- /tmp
14+
links:
15+
- postgres_generic_bird
16+
restart: always
17+
18+
postgres_generic_bird:
19+
image: ${GENERIC_BIRD_POSTGRES_IMAGE:-postgres:10.12}
20+
container_name: postgres_generic_bird
21+
volumes:
22+
- postgres_generic_bird:/var/lib/postgresql/data/pgdata
23+
- ./optional-components/generic_bird/postgres/docker-entrypoint-initdb.d:/docker-entrypoint-initdb.d
24+
environment:
25+
PGDATA: /var/lib/postgresql/data/pgdata
26+
env_file:
27+
- ./config/postgres/credentials.env
28+
restart: always
29+
30+
magpie:
31+
volumes:
32+
- ./optional-components/generic_bird/generic_bird-magpie-permission.cfg:/opt/local/src/magpie/config/permissions/generic_bird-magpie-permission.cfg:ro
33+
- ./optional-components/generic_bird/generic_bird-magpie-provider.cfg:/opt/local/src/magpie/config/providers/generic_bird-magpie-provider.cfg:ro
34+
35+
proxy:
36+
volumes:
37+
- ./optional-components/generic_bird/generic_bird_canarie_api_monitoring.py:${CANARIE_MONITORING_EXTRA_CONF_DIR}/generic_bird_canarie_api_monitoring.py:ro
38+
39+
volumes:
40+
postgres_generic_bird: {}

‎birdhouse/optional-components/generic_bird/generic_bird-magpie-permission.cfg.template

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
permissions:
2+
- service: ${GENERIC_BIRD_NAME}
3+
permission: getcapabilities
4+
group: anonymous
5+
action: create
6+
7+
- service: ${GENERIC_BIRD_NAME}
8+
permission: describeprocess
9+
group: anonymous
10+
action: create
11+
12+
- service: ${GENERIC_BIRD_NAME}
13+
permission: execute
14+
group: anonymous
15+
action: create

‎birdhouse/optional-components/generic_bird/generic_bird-magpie-provider.cfg.template

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
providers:
2+
${GENERIC_BIRD_NAME}:
3+
url: http://${PAVICS_FQDN}:${GENERIC_BIRD_PORT}/wps
4+
title: ${GENERIC_BIRD_NAME}
5+
public: true
6+
c4i: false
7+
type: wps
8+
sync_type: wps

‎birdhouse/optional-components/generic_bird/generic_bird_canarie_api_monitoring.py.template

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
SERVICES['node']['monitoring'].update({
2+
'${GENERIC_BIRD_NAME}-public': {
3+
'request': {
4+
'url': 'https://${PAVICS_FQDN_PUBLIC}/${TWITCHER_PROTECTED_PATH}/${GENERIC_BIRD_NAME}?service=WPS&version=1.0.0&request=GetCapabilities'
5+
},
6+
},
7+
'${GENERIC_BIRD_NAME}': {
8+
'request': {
9+
'url': 'http://${PAVICS_FQDN}:${GENERIC_BIRD_PORT}/wps?service=WPS&version=1.0.0&request=GetCapabilities'
10+
}
11+
},
12+
})
13+
14+
# vi: tabstop=8 expandtab shiftwidth=4 softtabstop=4 syntax=python

‎birdhouse/optional-components/generic_bird/postgres/.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
credentials.env

‎birdhouse/optional-components/generic_bird/postgres/credentials.env.template

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
POSTGRES_USER=${POSTGRES_PAVICS_USERNAME}
2+
POSTGRES_PASSWORD=${POSTGRES_PAVICS_PASSWORD}
3+
POSTGRES_DB=pavics

‎birdhouse/optional-components/generic_bird/postgres/docker-entrypoint-initdb.d/create-wps-databases.sql

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
CREATE DATABASE generic_bird;

‎birdhouse/optional-components/generic_bird/wps.cfg.template

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
[server]
2+
outputurl = https://${PAVICS_FQDN_PUBLIC}/wpsoutputs
3+
outputpath = /data/wpsoutputs
4+
5+
[logging]
6+
level = INFO
7+
database=postgresql://${POSTGRES_PAVICS_USERNAME}:${POSTGRES_PAVICS_PASSWORD}@postgres_generic_bird/generic_bird
8+
9+
${EXTRA_PYWPS_CONFIG}

‎birdhouse/pavics-compose.sh

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,8 @@ OPTIONAL_VARS='
4646
$AUTODEPLOY_PLATFORM_FREQUENCY
4747
$AUTODEPLOY_NOTEBOOK_FREQUENCY
4848
$AUTODEPLOY_EXTRA_SCHEDULER_JOBS
49+
$GENERIC_BIRD_PORT
50+
$GENERIC_BIRD_NAME
4951
'
5052

5153
# we switch to the real directory of the script, so it still works when used from $PATH

‎birdhouse/scripts/clear-running-wps-jobs-in-db.sh

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,8 +15,12 @@ else
1515
shift
1616
fi
1717

18+
if [ -z "$POSTGRES_CONTAINER_NAME" ]; then
19+
POSTGRES_CONTAINER_NAME=postgres
20+
fi
21+
1822
set -x
19-
docker exec postgres psql -U $POSTGRES_USER $DB_NAME -c "select * from pywps_requests where percent_done > -1 and percent_done < 100.0;"
23+
docker exec $POSTGRES_CONTAINER_NAME psql -U $POSTGRES_USER $DB_NAME -c "select * from pywps_requests where percent_done > -1 and percent_done < 100.0;"
2024

2125
set +x
2226
echo "
@@ -27,4 +31,4 @@ Clear those jobs? (Ctrl-C to cancel, any keys to continue)"
2731
read a
2832

2933
set -x
30-
docker exec postgres psql -U $POSTGRES_USER $DB_NAME -c "delete from pywps_requests where percent_done > -1 and percent_done < 100.0;"
34+
docker exec $POSTGRES_CONTAINER_NAME psql -U $POSTGRES_USER $DB_NAME -c "delete from pywps_requests where percent_done > -1 and percent_done < 100.0;"

0 commit comments

Comments
 (0)