Merge pull request #330 from bird-house/setup-cruft-for-cookiecutter-refresh

Setup cruft for cookiecutter refresh.

This PR is part of issue bird-house/cookiecutter-birdhouse#61, rolling out cruft as a mechanism to easily update cookiecutter template changes to existing project.

Require cookiecutter-birdhouse PR bird-house/cookiecutter-birdhouse#89

In `.cruft.json`, I set `"project_short_description": "A Web Processing Service Testbed."` from the `README.rst`, let me know if that is still correct.

In `README.rst`, I took the Flyingpigeon bird description from `docs/source/index.rst`.

Noticeable changes:

* Notebooks testing enabled in Travis-CI but allowed to fail, waiting for this issue #328 according to @nilshempelmann 
* Fix pymetalink install to be able to run notebook tests
* Docs testing also enabled in Travis-CI

Travis-CI build working: https://travis-ci.org/github/bird-house/flyingpigeon/builds/694013491
RtD build working: https://readthedocs.org/projects/flyingpigeon/builds/11184473/

Author: tlvu

.cruft.json

@@ -0,0 +1,36 @@
+{
+    "template": "https://github.com/bird-house/cookiecutter-birdhouse.git",
+    "commit": "fce99f3071d65bb494a44f7527b5dc5d92fbd390",
+    "skip": [
+        "flyingpigeon/processes/wps_say_hello.py",
+        "tests/test_wps_hello.py",
+        "docs/source/notebooks/example.ipynb",
+        "environment.yml",
+        "requirements.txt",
+        "AUTHORS.rst",
+        "CHANGES.rst",
+        "docs/source/notebooks/index.rst",
+        "docs/source/processes.rst",
+        "flyingpigeon/processes/__init__.py",
+        "tests/test_wps_caps.py"
+    ],
+    "context": {
+        "cookiecutter": {
+            "full_name": "Nils Hempelmann",
+            "email": "[email protected]",
+            "github_username": "bird-house",
+            "project_name": "Flyingpigeon",
+            "project_slug": "flyingpigeon",
+            "project_repo_name": "flyingpigeon",
+            "project_readthedocs_name": "flyingpigeon",
+            "project_short_description": "A Web Processing Service Testbed.",
+            "version": "1.5.1",
+            "open_source_license": "Apache Software License 2.0",
+            "http_port": "8093",
+            "_copy_without_render": [
+                "{{cookiecutter.project_slug}}/templates/*.cfg"
+            ],
+            "_template": "https://github.com/bird-house/cookiecutter-birdhouse.git"
+        }
+    }
+}

.github/CONTRIBUTING.md

@@ -2,5 +2,5 @@
 
 Contributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given.
 
-Please read the Birdhouse [Contributer Guide](http://birdhouse.readthedocs.io/en/latest/contributing.html)
-and the [Cookiecutter Documentation](http://flyingpigeon.readthedocs.io/en/latest/) to get started.
+Please read the Birdhouse [Developer Guide](https://birdhouse.readthedocs.io/en/latest/dev_guide.html)
+and the [Flyingpigeon Documentation](https://flyingpigeon.readthedocs.io/en/latest/) to get started.

.gitignore

@@ -28,7 +28,6 @@ __pycache__
 .tox
 nosetests.xml
 unit_tests/testdata.json
-docs/source/output_sanitize.cfg
 
 # R
 *.Rhistory
@@ -69,6 +68,7 @@ docs/make.bat
 docs/doctrees/
 docs/html/
 docs/build/
+docs/source/output-sanitize.cfg
 
 # External Sources
 #src/external
@@ -88,5 +88,8 @@ testdata.json
 *.mod
 *.out
 
+# Merge conflict
+*.orig
+
 # custom
 etc/custom.cgf

.travis.yml

@@ -1,38 +1,44 @@
 dist: xenial   # required for Python >= 3.7
 language: python
+os:
+  - linux
 python:
   - "3.6"
   - "3.7"
 branches:
   only:
     - master
-os:
-  - linux
 #  - osx
 sudo: false
+before_install:
+  # Useful for debugging Travis CI environment
+  - printenv
 install:
   # Python 3.x is default
   - wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
   - bash miniconda.sh -b -p $HOME/miniconda
   - export PATH="$HOME/miniconda/bin:$PATH"
   - hash -r
   - conda config --set always_yes yes --set changeps1 no
+  - conda install setuptools
   - conda update -q conda
   # Useful for debugging any issues with conda
   - conda info -a
   # Prepare env with Python version
-  - conda create -n flyingpigeon python=$TRAVIS_PYTHON_VERSION
+  - conda create -n flyingpigeon -c conda-forge python=$TRAVIS_PYTHON_VERSION
   # Update now the env with our environment
   - conda env update -f environment.yml
   - source activate flyingpigeon
-  # Packages for testing
-  - conda install pytest flake8
+  # Packages for testing and generating docs
+  - make develop
   # Install WPS
   - python setup.py install
 before_script:
-  # Start WPS service on port 5000 on 0.0.0.0
-  - flyingpigeon start --daemon --bind-host 0.0.0.0 --port 5000
+  # Start WPS service on port 8093 on 0.0.0.0
+  - flyingpigeon start --daemon --bind-host 0.0.0.0 --port 8093
   - sleep 2
 script:
   - pytest
+  - make test-notebooks || echo "allow fail"
   - flake8
+  - make docs

Makefile

@@ -5,6 +5,9 @@ APP_NAME := flyingpigeon
 # TODO read in from server configuration
 WPS_URL = http://localhost:8093
 OUTPUT_URL = https://pavics.ouranos.ca/wpsoutputs
+
+SANITIZE_FILE := https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/raw/master/notebooks/output-sanitize.cfg
+
 # end of configuration
 
 .DEFAULT_GOAL := help
@@ -26,7 +29,9 @@ help:
 	@echo "\nTesting targets:"
 	@echo "  test              to run tests (but skip long running tests)."
 	@echo "  test-all          to run all tests (including long running tests)."
+	@echo "  test-notebooks    to verify Jupyter Notebook test outputs are valid."
 	@echo "  lint              to run code style checks with flake8."
+	@echo "  refresh-notebooks to verify Jupyter Notebook test outputs are valid."
 	@echo "\nSphinx targets:"
 	@echo "  docs              to generate HTML documentation with Sphinx."
 	@echo "\nDeployment targets:"
@@ -44,6 +49,7 @@ install:
 develop:
 	@echo "Installing development requirements for tests and docs ..."
 	@-bash -c 'pip install -e ".[dev]"'
+	@-bash -c 'pip install git+https://github.com/metalink-dev/[email protected]#egg=pymetalink --upgrade'
 
 .PHONY: start
 start:
@@ -61,15 +67,15 @@ restart: stop start
 
 .PHONY: status
 status:
-	@echo "Show status ..."
+	@echo "Showing status ..."
 	@-bash -c "$(APP_NAME) status"
 
 .PHONY: clean
 clean: clean-build clean-pyc clean-test ## remove all build, test, coverage and Python artifacts
 
 .PHONY: clean-build
 clean-build:
-	@echo "Remove build artifacts ..."
+	@echo "Removing build artifacts ..."
 	@-rm -fr build/
 	@-rm -fr dist/
 	@-rm -fr .eggs/
@@ -80,21 +86,22 @@ clean-build:
 
 .PHONY: clean-pyc
 clean-pyc:
-	@echo "Remove Python file artifacts ..."
+	@echo "Removing Python file artifacts ..."
 	@-find . -name '*.pyc' -exec rm -f {} +
 	@-find . -name '*.pyo' -exec rm -f {} +
 	@-find . -name '*~' -exec rm -f {} +
 	@-find . -name '__pycache__' -exec rm -fr {} +
 
 .PHONY: clean-test
 clean-test:
-	@echo "Remove test artifacts ..."
+	@echo "Removing test artifacts ..."
 	@-rm -fr .pytest_cache
 
 .PHONY: clean-dist
 clean-dist: clean
-	@echo "Run 'git clean' ..."
-	@git diff --quiet HEAD || echo "There are uncommited changes! Not doing 'git clean' ..."
+	@echo "Running 'git clean' ..."
+	@git diff --quiet HEAD || echo "There are uncommitted changes! Aborting 'git clean' ..."
+	## do not use git clean -e/--exclude here, add them to .gitignore instead
 	@-git clean -dfx
 
 ## Test targets
@@ -109,37 +116,41 @@ test-all:
 	@echo "Running all tests (including slow and online tests) ..."
 	@bash -c 'pytest -v tests/'
 
+.PHONY: notebook-sanitizer
+notebook-sanitizer:
+	@echo "Copying notebook output sanitizer ..."
+	@-bash -c "curl -L $(SANITIZE_FILE) -o $(CURDIR)/docs/source/output-sanitize.cfg --silent"
+
 .PHONY: test-notebooks
-test-notebooks:
+test-notebooks: notebook-sanitizer
 	@echo "Running notebook-based tests"
-	@bash -c "curl -L https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/raw/master/notebooks/output-sanitize.cfg --output $(CURDIR)/docs/source/output_sanitize.cfg --silent"
-	@bash -c "env WPS_URL=$(WPS_URL) pytest --nbval --verbose $(CURDIR)/docs/source/notebooks/ --sanitize-with $(CURDIR)/docs/source/output_sanitize.cfg --ignore $(CURDIR)/docs/source/notebooks/.ipynb_checkpoints"
-
-.PHONY: refresh-notebooks
-refresh-notebooks:
-	@echo "Refresh all notebook outputs under docs/source/notebooks"
-	cd docs/source/notebooks; for nb in *.ipynb; do WPS_URL="$(WPS_URL)" jupyter nbconvert --to notebook --execute --ExecutePreprocessor.timeout=60 --output "$$nb" "$$nb"; sed -i "s@$(WPS_URL)/outputs/@$(OUTPUT_URL)/@g" "$$nb"; done; cd $(APP_ROOT)
-
-
+	@bash -c "env WPS_URL=$(WPS_URL) pytest --nbval --verbose $(CURDIR)/docs/source/notebooks/ --sanitize-with $(CURDIR)/docs/source/output-sanitize.cfg --ignore $(CURDIR)/docs/source/notebooks/.ipynb_checkpoints"
 
 .PHONY: lint
 lint:
 	@echo "Running flake8 code style checks ..."
 	@bash -c 'flake8'
 
+.PHONY: refresh-notebooks
+refresh-notebooks:
+	@echo "Refresh all notebook outputs under docs/source/notebooks"
+	@bash -c 'for nb in $(CURDIR)/docs/source/notebooks/*.ipynb; do WPS_URL="$(WPS_URL)" jupyter nbconvert --to notebook --execute --ExecutePreprocessor.timeout=60 --output "$$nb" "$$nb"; sed -i "s@$(WPS_URL)/outputs/@$(OUTPUT_URL)/@g" "$$nb"; done; cd $(APP_ROOT)'
+
 ## Sphinx targets
 
 .PHONY: docs
 docs:
 	@echo "Generating docs with Sphinx ..."
 	@-bash -c '$(MAKE) -C $@ clean html'
-	@echo "open your browser: open file://$(APP_ROOT)/docs/build/html/index.html"
+	@echo "Open your browser to: file:/$(APP_ROOT)/docs/build/html/index.html"
+	## do not execute xdg-open automatically since it hangs travis and job does not complete
+	@echo "xdg-open $(APP_ROOT)/docs/build/html/index.html"
 
 ## Deployment targets
 
 .PHONY: dist
 dist: clean
-	@echo "Builds source and wheel package ..."
+	@echo "Building source and wheel package ..."
 	@-python setup.py sdist
 	@-python setup.py bdist_wheel
-	ls -l dist
+	@-bash -c 'ls -l dist/'

README.rst

@@ -18,17 +18,45 @@ Flyingpigeon
     :alt: Join the chat at https://gitter.im/bird-house/birdhouse
 
 
-A Web Processing Service Testbed
---------------------------------
+Flyingpigeon (the bird)
+  *The pigeon finds its way home over extremely long distances. [..].* ( `Wikipedia <https://en.wikipedia.org/wiki/Pigeon_flying>`_ ).
+
+A Web Processing Service Testbed.
 
 Flyingpigeon is a server providing a sandbox and test environment for new Web Processing Services (OGC WPS). It is part of the `birdhouse`_ ecosystem, which aims to build interoperable services in support to substainable development goals. Once stable, mature and thoroughly tested, Flyingpigeon services are meant to move to stand-alone thematic servers used in production.
 
 The Flyingpigeon software stack as of version 1.0 was published in `Computers & Geosciences <https://www.sciencedirect.com/science/article/pii/S0098300416302801>`_ and can still be accessed from the `release page <https://github.com/bird-house/flyingpigeon/releases/tag/1.0.0>`_. Meanwhile many of the 1.0 processes have migrated to other birdhouse repositories. For example, processes related to extreme weather event assessment can be found in `blackswan`_, while processes focusing on climate indices will be found in `finch`_.
 
-* Documentation: https://flyingpigeon.readthedocs.io
-* GitHub repository: https://github.com/bird-house/flyingpigeon
-* Birdhouse overview: https://birdhouse.readthedocs.io/en/latest/
+Documentation
+-------------
+
+Learn more about Flyingpigeon in its official documentation at
+https://flyingpigeon.readthedocs.io.
+
+Submit bug reports, questions and feature requests at
+https://github.com/bird-house/flyingpigeon/issues
+
+Contributing
+------------
+
+You can find information about contributing in our `Developer Guide`_.
+
+Please use bumpversion_ to release a new version.
+
+License
+-------
+
+Free software: Apache Software License 2.0
+
+Credits
+-------
+
+This package was created with Cookiecutter_ and the `bird-house/cookiecutter-birdhouse`_ project template.
 
+.. _Cookiecutter: https://github.com/audreyr/cookiecutter
+.. _`bird-house/cookiecutter-birdhouse`: https://github.com/bird-house/cookiecutter-birdhouse
+.. _`Developer Guide`: https://flyingpigeon.readthedocs.io/en/latest/dev_guide.html
+.. _bumpversion: https://flyingpigeon.readthedocs.io/en/latest/dev_guide.html#bump-a-new-version
 .. _blackswan: https://github.com/bird-house/blackswan
 .. _birdhouse: https://birdhouse.readthedocs.io/en/latest/
 .. _finch: https://github.com/bird-house/finch