Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
7b836f8
fix CanarieAPI config to support Python 3.x
fmigneault Jan 31, 2023
340106c
update changes
fmigneault Jan 31, 2023
50706db
adjust exec_file to consider keyword arguments and Python 2/3 simulta…
fmigneault Feb 2, 2023
8e27f7f
update changelog about canarieapi Python 3 configs requirement
fmigneault Feb 3, 2023
a12d028
add metadata for all services represented under canarie api
fmigneault Feb 3, 2023
76743a4
fixes to canarieapi configs
fmigneault Feb 3, 2023
ed2a987
Merge branch 'master' into fix-canarieapi
fmigneault Feb 7, 2023
c0e8b38
Merge branch 'master' into fix-canarieapi
fmigneault Feb 10, 2023
68ec0b2
Merge branch 'master' into fix-canarieapi
fmigneault Feb 15, 2023
80084fd
drop python 2 specific code in canarieapi config
fmigneault Feb 16, 2023
8a51f9d
add HTTP repsonse body to raised error for more context if provided b…
fmigneault Feb 16, 2023
c3191e7
Merge branch 'fix-canarieapi' of https://github.com/bird-house/birdho…
fmigneault Feb 16, 2023
5fec973
move weaver public endpoint under canarieapi full monitoring + add mo…
fmigneault Feb 17, 2023
eb4a0d4
pin canarieapi 0.5.0 for latest Python 3.11 version for security fixe…
fmigneault Feb 17, 2023
cb6dfa6
revert ports definitions in canarieapi config (relates to https://git…
fmigneault Feb 17, 2023
77642e3
add request caching for loading canarieapi config
fmigneault Feb 17, 2023
ae7da73
revert tmp dir for security reasons
fmigneault Feb 17, 2023
b0808ef
update CanarieAPI 0.5.1
fmigneault Feb 17, 2023
811569a
Merge remote-tracking branch 'origin/master' into fix-canarieapi
fmigneault Feb 17, 2023
278dcfb
use canarieapi cached requests for tag version/release-time check by …
fmigneault Feb 20, 2023
69189ac
Merge remote-tracking branch 'origin/master' into fix-canarieapi
fmigneault Mar 22, 2023
669674d
fix changelog after merge conflict
fmigneault Mar 22, 2023
9b50299
allow public access to cowbird version endpoint
fmigneault Mar 22, 2023
0bf7dfc
Merge branch 'master' into fix-canarieapi
mishaschwartz Mar 23, 2023
e50de77
a few additional fixes after the merge with master
mishaschwartz Mar 23, 2023
233ca09
update canarie-api to version 0.6.0
mishaschwartz Mar 23, 2023
e8df40b
small fixes: fix phoenix monitoring url protocol and fix missing giti…
mishaschwartz Mar 23, 2023
587a7ef
Merge branch 'fix-canarieapi' of https://github.com/bird-house/birdho…
fmigneault Mar 23, 2023
52532f6
canarie-api-full-monitoring implies all-public access
mishaschwartz Mar 24, 2023
160a6ea
hacky fix to get solr working with canarie again (will remove this sh…
mishaschwartz Mar 24, 2023
b355a37
Merge branch 'master' into fix-canarieapi
fmigneault Mar 27, 2023
f49db35
Merge branch 'master' into fix-canarieapi
mishaschwartz Apr 10, 2023
4ccab38
Merge branch 'master' into fix-canarieapi
mishaschwartz Apr 11, 2023
f072d21
update thredds version to one that exists
mishaschwartz Apr 11, 2023
3dc83f2
Merge branch 'master' into fix-canarieapi
mishaschwartz Apr 12, 2023
1d0dd73
avoid github response setting 60s for canarie-api request cache expir…
fmigneault Apr 14, 2023
fa427a9
update canarieapi 0.7.0 to fix issue https://github.com/bird-house/bi…
fmigneault Apr 14, 2023
990e1f3
update canarieapi 0.7.1 to fix some security issues flagged by docker…
fmigneault Apr 17, 2023
bdde8c8
Merge branch 'master' into fix-canarieapi
fmigneault Apr 18, 2023
59baaa7
update changes with canarieapi ui image
fmigneault Apr 18, 2023
34c02fb
fix catalog doc URL in canarieapi config
fmigneault Apr 18, 2023
a4784b9
fix jupyterhub doc URL in canarieapi config
fmigneault Apr 18, 2023
0a1b259
retrieve Twitcher version/release time from Magpie dockerfile tag for…
fmigneault Apr 18, 2023
08041e6
add missing canarieapi configs for flyingepigeon, hummingbird + use v…
fmigneault Apr 18, 2023
2283efd
update unittest with missing canarieapi configs newly added
fmigneault Apr 18, 2023
d0ebf98
fix typo in repo location for flyingpigeon/hummingbird
fmigneault Apr 19, 2023
9fced0d
add gitignore entries for new canarieapi configs
fmigneault Apr 19, 2023
eea7230
remove extra comma causing tuple as release time in canarieapi configs
fmigneault Apr 19, 2023
a28bafe
Merge branch 'master' into fix-canarieapi
fmigneault Apr 19, 2023
f4da986
update hummingbird canarieapi config reference for actual tag location
fmigneault Apr 19, 2023
ede869c
Merge remote-tracking branch 'origin/master' into fix-canarieapi
fmigneault Apr 20, 2023
e32baa7
revert all-public-access dependency under canarie-api-full-monitoring
fmigneault Apr 20, 2023
5bbf294
Bump version: 1.25.7 → 1.26.0
fmigneault Apr 20, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .bumpversion.cfg
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[bumpversion]
current_version = 1.25.7
current_version = 1.26.0
commit = True
tag = False
tag_name = {new_version}
Expand Down Expand Up @@ -30,11 +30,11 @@ search = {current_version}
replace = {new_version}

[bumpversion:file:RELEASE.txt]
search = {current_version} 2023-04-20T14:15:34Z
search = {current_version} 2023-04-20T20:42:31Z
replace = {new_version} {utcnow:%Y-%m-%dT%H:%M:%SZ}

[bumpversion:part:releaseTime]
values = 2023-04-20T14:15:34Z
values = 2023-04-20T20:42:31Z

[bumpversion:file(version):birdhouse/config/canarie-api/docker_configuration.py.template]
search = 'version': '{current_version}'
Expand Down
58 changes: 58 additions & 0 deletions CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,64 @@

[//]: # (list changes here, using '-' for each new entry, remove this when items are added)

[1.26.0](https://github.com/bird-house/birdhouse-deploy/tree/1.26.0) (2023-04-20)
------------------------------------------------------------------------------------------------------------------


## Breaking changes

- CanarieAPI: update to `0.7.1`.

- The Docker running `CanarieAPI` is now using Python 3 (since `0.4.x` tags).
Configurations need to be updated if any specific Python 2 definitions were used.
See [2to3](https://docs.python.org/3/library/2to3.html) to help migrate configurations automatically if necessary.
- Update the [CanarieAPI configuration](birdhouse/config/canarie-api/docker_configuration.py.template) to use
Python 3.x executable code.

## Changes

- CanarieAPI: update to `0.7.1`.

- The server node now provides a generic ``server`` configuration for the current ``platform`` definition.
- Added multiple missing docuementation references for all the services included within `CanarieAPI` configurations.
- With new `CanarieAPI` version, a slightly improved UI with more service details are provided for the active server:

![image](https://user-images.githubusercontent.com/19194484/232822454-e39c0111-54dc-4f9b-adf6-5ea6e59d67e3.png)

- Add optional variables witht defaults to define reference Docker image version tags.

Following optional variables are defined by default. These are used as reference in the respective Docker compose
service definition of these components, as well as in their `CanarieAPI` configuration to retrieve the release time
of the tag, and refer to relevant URL references as needed.

- `CATALOG_VERSION`
- `FINCH_VERSION`
- `FLYINGPIGEON_VERSION`
- `GEOSERVER_VERSION`
- `HUMMINGBIRD_VERSION`
- `MALLEEFOWL_VERSION`
- `RAVEN_VERSION`

## Fixes:

- CanarieAPI: update to `0.7.1`.

- Fixes an `AttributeError` raised due to misconfiguration of the Web Application with Flask 2.x definitions
(relates to [Ouranosinc/CanarieAPI#10](https://github.com/Ouranosinc/CanarieAPI/pull/10)).
- Skip over `0.4.x`, `0.5.x`, `0.6.x` versions to avoid issue related to `cron` job monitoring and log parser
command failures in order to collect configured service statistics and statuses
(see also [Ouranosinc/CanarieAPI#14](https://github.com/Ouranosinc/CanarieAPI/pull/14)).

- Weaver: update CanarieAPI monitoring definitions
- Move monitoring of public endpoint under [optional-components/canarie-api-full-monitoring][canarie-monitor].
- Add monitoring of private endpoint by default when using Weaver component.

- Cowbird: update CanarieAPI monitoring definitions
- Add monitoring of public endpoint under [optional-components/canarie-api-full-monitoring][canarie-monitor].
- Add public Magpie permission on Cowbird entrypoint only to allow its monitoring.

[canarie-monitor]: birdhouse/optional-components/canarie-api-full-monitoring

[1.25.7](https://github.com/bird-house/birdhouse-deploy/tree/1.25.7) (2023-04-20)
------------------------------------------------------------------------------------------------------------------

Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Generic variables
override SHELL := bash
override APP_NAME := birdhouse-deploy
override APP_VERSION := 1.25.7
override APP_VERSION := 1.26.0

# utility to remove comments after value of an option variable
override clean_opt = $(shell echo "$(1)" | $(_SED) -r -e "s/[ '$'\t'']+$$//g")
Expand Down
8 changes: 4 additions & 4 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,13 @@ for a full-fledged production platform.
* - releases
- | |latest-version| |commits-since|

.. |commits-since| image:: https://img.shields.io/github/commits-since/bird-house/birdhouse-deploy/1.25.7.svg
.. |commits-since| image:: https://img.shields.io/github/commits-since/bird-house/birdhouse-deploy/1.26.0.svg
:alt: Commits since latest release
:target: https://github.com/bird-house/birdhouse-deploy/compare/1.25.7...master
:target: https://github.com/bird-house/birdhouse-deploy/compare/1.26.0...master

.. |latest-version| image:: https://img.shields.io/badge/tag-1.25.7-blue.svg?style=flat
.. |latest-version| image:: https://img.shields.io/badge/tag-1.26.0-blue.svg?style=flat
:alt: Latest Tag
:target: https://github.com/bird-house/birdhouse-deploy/tree/1.25.7
:target: https://github.com/bird-house/birdhouse-deploy/tree/1.26.0

.. |readthedocs| image:: https://readthedocs.org/projects/birdhouse-deploy/badge/?version=latest
:alt: ReadTheDocs Build Status (latest version)
Expand Down
2 changes: 1 addition & 1 deletion RELEASE.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
1.25.7 2023-04-20T14:15:34Z
1.26.0 2023-04-20T20:42:31Z
2 changes: 2 additions & 0 deletions birdhouse/components/cowbird/.gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ config/cowbird/config.yml
config/cowbird/cowbird.ini
config/magpie/config.yml
config/proxy/conf.extra-service.d/cowbird.conf
config/proxy/canarie_api_monitoring.py

# Old paths. Keep these so that old config files remain uncommittable after updates.
conf.extra-service.d/cowbird.conf
config/canarie-api/cowbird_config.py
11 changes: 11 additions & 0 deletions birdhouse/components/cowbird/config/magpie/config.yml.template
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,17 @@ permissions:
permission: write
group: administrators
action: create
# Allow validating/monitoring the service entrypoint and its version
# Everything else should remain admin-protected since it manages other user/group permissions.
- service: cowbird
permission: read-match
group: anonymous
action: create
- service: cowbird
resource: /version
permission: read-match
group: anonymous
action: create

# Definitions of all the webhooks urls that will be called when creating or deleting a user.
webhooks:
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
SERVICES['Cowbird'] = {
'info': {
'name': 'Cowbird',
'synopsis': 'Cowbird is a middleware that manages interactions between various birds of the bird-house stack.',
'version': "${COWBIRD_VERSION}",
'institution': 'Ouranos, CRIM',
'releaseTime': get_release_time_from_repo_tag("github", "Ouranosinc/cowbird", "${COWBIRD_VERSION}"),
'researchSubject': 'Any',
'supportEmail': '${SUPPORT_EMAIL}',
'category': 'Security',
'tags': ['Security', 'Management', 'Access', 'Policy Decision Point']
},
'stats': {
'method': '.*',
'route': "/cowbird/.*"
},
'redirect': {
'doc': 'https://pavics-cowbird.readthedocs.io/',
'releasenotes': 'https://github.com/Ouranosinc/cowbird//blob/master/CHANGES.rst',
'support': 'https://github.com/Ouranosinc/cowbird//issues',
'source': 'https://github.com/Ouranosinc/cowbird/',
'tryme': 'https://${PAVICS_FQDN_PUBLIC}/cowbird/',
'licence': 'https://github.com/Ouranosinc/cowbird//blob/${COWBIRD_VERSION}/LICENSE',
'provenance': 'https://github.com/Ouranosinc/cowbird/'
},
"monitoring": {
"Cowbird": {
'request': {
'url': 'http://cowbird:7000/'
},
'response': {
'text': '\{.*"title":.*"Cowbird REST API".*\}'
}
}
}
}

# vi: tabstop=8 expandtab shiftwidth=4 softtabstop=4 syntax=python
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,12 @@ services:
proxy:
volumes:
- ./components/cowbird/config/proxy/conf.extra-service.d:/etc/nginx/conf.extra-service.d/cowbird:ro
# NOTE:
# Prefix '0_' to the mounted file name to ensure it is loaded first by 'birdhouse/config/canarie-api/docker_configuration.py'
# This ensures that the alphabetical loading order it defines will first load the default configs, then will load
# the Cowbird config, and then all python configuration within 'optional-components/canarie-api-full-monitoring'.
# It is important to load Cowbird before, as the full monitoring need to know if 'Cowbird' service is added to the
# stack in order to apply the corresponding public endpoint conditionally.
- ./components/cowbird/config/proxy/canarie_api_monitoring.py:${CANARIE_MONITORING_EXTRA_CONF_DIR}/0_cowbird_config.py:ro
links:
- cowbird
Original file line number Diff line number Diff line change
@@ -1,9 +1,38 @@
SERVICES['node']['monitoring'].update({
'Weaver': {
'request': {
'url': 'https://${PAVICS_FQDN_PUBLIC}/weaver'
},
SERVICES['Weaver'] = {
'info': {
'name': 'Weaver',
'synopsis': 'Weaver is an OGC-API flavored Execution Management Service (EMS) that allows the execution of workflows chaining various applications and Web Processing Services (WPS) inputs and outputs. Remote execution is deferred by the EMS to one or many Application, Deployment and Execution Service (ADES) or remote service providers, and employs Common Workflow Language (CWL) configurations to define an OGC Application Package deployed for each process.',
'version': "${WEAVER_VERSION}",
'institution': 'CRIM',
'releaseTime': get_release_time_from_repo_tag("github", "crim-ca/weaver", "${WEAVER_VERSION}"),
'researchSubject': 'Any',
'supportEmail': '${SUPPORT_EMAIL}',
'category': 'Processing',
'tags': ['Workflow', 'Processes', 'Deployment', 'Distribution', 'EMS', 'ADES', 'CWL', 'OGC']
},
})
'stats': {
'method': '.*',
'route': "/weaver/.*"
},
'redirect': {
'doc': 'https://pavics-weaver.readthedocs.io/',
'releasenotes': 'https://github.com/crim-ca/weaver/blob/master/CHANGES.rst',
'support': 'https://github.com/crim-ca/weaver/issues',
'source': 'https://github.com/crim-ca/weaver',
'tryme': 'https://${PAVICS_FQDN_PUBLIC}/weaver/',
'licence': 'https://github.com/crim-ca/weaver/blob/${WEAVER_VERSION}/LICENSE.txt',
'provenance': 'https://github.com/crim-ca/weaver'
},
"monitoring": {
"Weaver": {
'request': {
'url': 'http://weaver:4001/'
},
'response': {
'text': '\{.*"message":.*"Weaver Information".*\}'
}
}
}
}

# vi: tabstop=8 expandtab shiftwidth=4 softtabstop=4 syntax=python
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,11 @@ services:
# extend proxy configuration with weaver endpoints
proxy:
volumes:
# NOTE:
# Prefix '0_' to the mounted file name to ensure it is loaded first by 'birdhouse/config/canarie-api/docker_configuration.py'
# This ensures that the alphabetical loading order it defines will first load the default configs, then will load
# the weaver config, and then all python configuration within 'optional-components/canarie-api-full-monitoring'.
# It is important to load Weaver before, as the full monitoring need to know if 'Weaver' service is added to the
# stack in order to apply the corresponding public endpoint conditionally.
- ./components/weaver/config/canarie-api/canarie_api_monitoring.py:${CANARIE_MONITORING_EXTRA_CONF_DIR}/0_weaver_config.py:ro
- ./components/weaver/config/canarie-api/canarie_api_monitoring.py:${CANARIE_MONITORING_EXTRA_CONF_DIR}/weaver_canarie_api_monitoring.py:ro
3 changes: 1 addition & 2 deletions birdhouse/config/canarie-api/default.env
Original file line number Diff line number Diff line change
@@ -1,10 +1,9 @@
# Folder inside "proxy" container to drop extra monitoring config
export CANARIE_MONITORING_EXTRA_CONF_DIR="/conf.d"

export PROXY_IMAGE="pavics/canarieapi:0.3.5"
export PROXY_IMAGE="pavics/canarieapi:0.7.1"

# add any component that this component requires to run
COMPONENT_DEPENDENCIES="
./config/proxy
"

Loading