From 0a8878aae06f74971d8f475c76cc3123b948d525 Mon Sep 17 00:00:00 2001 From: regularfry Date: Wed, 13 Sep 2023 19:42:50 +0100 Subject: [PATCH 1/6] Rewritten tutorial to write a very simple AWS Lambda The prior tutorial branch involved Docker usage that we don't want to involve as a first contact with the framework. This (minor) rewrite skips the Docker setup, instead leaning on `asdf` for python version management. --- Makefile | 34 ++- docs/tutorial.md | 396 +++++++++++++++++++++++++++++++ scripts/config/markdownlint.yaml | 6 + 3 files changed, 429 insertions(+), 7 deletions(-) create mode 100644 docs/tutorial.md diff --git a/Makefile b/Makefile index 2c893b23..4b497438 100644 --- a/Makefile +++ b/Makefile @@ -1,17 +1,25 @@ # This file is for you! Edit it to implement your own hooks (make targets) into # the project as automated steps to be executed on locally and in the CD pipeline. -include scripts/init.mk - -# ============================================================================== +include ./scripts/init.mk # Example CI/CD targets are: dependencies, build, publish, deploy, clean, etc. -dependencies: # Install dependencies needed to build and test the project @Pipeline +dependencies: # Install dependencies needed to build and test the project # TODO: Implement installation of your project dependencies -build: # Build the project artefact @Pipeline - # TODO: Implement the artefact build step + +build: # Build the project artefact @Pipeline + make _project name="build" + +up: # Run your code + make _project name="up" + +down: # Stop your code + make _project name="down" + +sh: up # Get a shell inside your running project, running it first if necessary + make _project name="sh" publish: # Publish the project artefact @Pipeline # TODO: Implement the artefact publishing step @@ -26,11 +34,23 @@ config:: # Configure development environment (main) @Configuration # TODO: Use only 'make' targets that are specific to this project, e.g. you may not need to install Node.js make _install-dependencies -# ============================================================================== +_project: + set -e + SCRIPT="./scripts/projecthooks/${name}.sh" + if [ -e "$${SCRIPT}" ]; then + exec $$SCRIPT + else + echo "make ${name} not implemented: $${SCRIPT} not found" >&2 + fi ${VERBOSE}.SILENT: \ + _project \ build \ clean \ config \ dependencies \ deploy \ + down \ + sh \ + tmp/build_timestamp \ + up \ diff --git a/docs/tutorial.md b/docs/tutorial.md new file mode 100644 index 00000000..ee5f5edf --- /dev/null +++ b/docs/tutorial.md @@ -0,0 +1,396 @@ +# The Simplest Thing That Can Possibly Work + +This repository template contains a lot of material, and provides a +lot of functionality out of the box. You might feel intimidated and +confused by the sheer number of files in it, and not know where to +look to get what you need done. + +This tutorial is to walk you through the steps of getting a basic +Hello-World-style app into production, to show you where to find the +functionality you need and to demystify some of the content so that +you can get the most out of it. + +We'll be building a python AWS Lambda handler, so to complete the +later parts of the tutorial you'll need AWS credentials that let you +deploy to it. + +I will assume familiarity with git itself. + +## First things first + +You will need certain tools to be installed in order to use the tools +in the repository template. They are: + +- The gnu coreutils. +- Gnu make. + +These are some of the tools that the engineering community assumes you +will have for compatibility with a wide range of projects and +products, so install them if you haven't already. + +Some other tools will be installed by the framework if you don't +already have them installed. I'll call those out as and when they +come up. + +I do not assume, dictate, or otherwite hint at the correct text +editor for you to be using. I just silently judge you for your +choice. + +## Cloning the template + +The first step to your brand new app is to clone the repository. For +the purposes of this tutorial, we're going to clone it into your +personal account rather than into any of the NHS github +organisations. That's not what you'd be doing for anything we want to +go into production, but it means you can keep it as a reference and we +don't need to worry about naming collisions. + +Go to in your +browser, and click the green button marked `Use this template`. From +the dropdown that appears, select `Create a new repository`. + +You will be taken to github's `Create a new repository` page. You +might notice that this is the same page as github shows you when you +create an ordinary repository in the web user interface, but pay +attention to the `Repository template` field: you will see that it is +pre-populated with the `nhs-england-tools/repository-template` value. +That is as we want, so leave it as is. + +Leave the `Include all branches` checkbox unticked. + +Under `Owner`, select yourself from the `Choose an owner` dropdown. +Visibility options will appear under the Description field. The +repository is MIT-licensed, and we're working in the open, so you +don't need to worry about this being a public repository. You can +make it private if you prefer. + +Under `Repository name`, give the repository a name of +`nhset-hello-world`. + +[TODO: This means the SonarCloud integration won't work, but we can't +get around that without being able to make repositories in a different +org.] + +If a description will help you remember the purpose of this project in +the future, put something meaningful to you in the `Description` +field. + +Click the green `Create repository` button, and github will show you a +loading page while your new instance of the repository is created. + +Now that you have your repository, clone it to your local machine for +the next step. + +## Python time + +For our Python application, we will be building an API endpoint which +returns a JSON blob that looks like this: + +```json +{ + "message": "Hello World" +} +``` + +The framework contains a mechanism for ensuring that your local +development python interpreter is at a known version; that mechanism +relies on [`asdf`](https://github.com/asdf-vm/asdf). The next step will +install it if you don't already have it, and will update it if you +do, but there's something we need to edit first. + +Open `Makefile` in your editor. This is the entrypoint for the tasks +that the framework supplies for you, and this entrypoint is something +you can edit for yourself. + +Locate the `config` task. You will see that it looks like this: + +```make +config:: # Configure development environment + # TODO: Use only `make` targets that are specific to this project, e.g. you may not need to install Node.js + make \ + nodejs-install \ + python-install \ + terraform-install +``` + +For this project we don't need NodeJS, and while we will need +terraform, we don't need it yet. Modify the `config` task so that it +looks like this: + +```make +config:: # Configure development environment + make python-install +``` + +That means it will only install python for us, which - for now - is +the only thing we want. Run the task: + +```shell +make config +``` + +The first time you run this it will download and build `python` for +you, so you may want to make a cup of tea. The version it picks is in +the `.tool-version` file at the top level of the template. + +In addition to installing `asdf`, `make config` sets up some `git` +commit hooks that will come in handy later. + +Now, the template has given us a `poetry` installation for dependency +management. There is a configuration option that I recommend you set +unless you have a good reason not to: + +```shell +poetry config virtualenvs.in-project true +``` + +This tells `poetry` to always make a virtual environment within the +project directory. If you don't set that, it's somewhere else - `poetry`'s +cache directory, to be specific. With it within the project's working +directory, fixing certain types of packaging problems becomes much +easier: if you get really stuck, you can be absolutely certain that +deleting the working directory and starting again will reset all +(well, nearly all) the relevant state. + +However, you don't want to be committing the virtual environment to +`git`, so edit the file `.gitignore` and add : + +```text +.venv/ +``` + +## A Failing Test + +Let's get ourselves set up to run a unit test. We'll want to use +`pytest` for this. The configuration we need for that lives in +`pyproject.toml`. That file doesn't exist yet, so edit it and add the +following: + +```toml +[tool.poetry] +name = "nhset-hello-world" +version = "2023.09.13" +description = "A short description" +authors = [] + +[tool.poetry.dependencies] +python = "^3.11" +``` + +Now we add `pytest`: + +```shell +poetry add --group=dev pytest +``` + +With that prologue out of the way, we can write our test. Which will +fail, but seeing it fail in the right way will tell us that the python +environment is how we need it. First, make a directory to put our +code into: + +```shell +mkdir api +``` + +Now, open `api/test_hello_world.py`, and add the following: + +```python +from hello_world import lambda_handler + +def test_lambda_handler(): + response = lambda_handler(None, None) + assert "message" in response + assert response["message"] == "Hello World" +``` + +Run pytest with the command `poetry run pytest` and you will see the +following (among some other test failure info): + +```console +$ poetry run pytest +... + from hello_world import lambda_handler +E ModuleNotFoundError: No module named 'hello_world' +... +``` + +So, edit `api/hello_world.py` and add: + +```python +def lambda_handler(event, context): + return {} +``` + +Run `poetry run pytest` again, and the output is a failure that we +should now be expecting: + +```text + def test_lambda_handler(): + response = lambda_handler(None, None) +> assert "message" in response +E AssertionError: assert 'message' in {} + +api/test_hello_world.py:5: AssertionError +``` + +Now let's get the test to pass. Edit the `lambda_handler` function in +`api/hello_world.py` to read as follows: + +```python +def lambda_handler(event, context): + return {"message": "Hello World"} +``` + +Running `pytest` again gives us what we want: + +```console + $ poetry run pytest --quiet +. [100%] +1 passed in 0.02s +``` + +In order to be able to integrate our tests with the rest of the +framework, we want to hide our `poetry run` behind a standard command +that the framework recognises. As you saw above, the template relies +on `make` for its entry points, and testing is no different. Run +`make test-unit` and you should see the following: + +```console + $ make test-unit +Unit tests are not yet implemented. See scripts/tests/unit.sh for more. +``` + +So, edit `scripts/tests/unit.sh` and change the last line from: + +```shell +echo "Unit tests are not yet implemented. See scripts/tests/unit.sh for more." +``` + +to + +```shell +poetry run pytest --quiet +``` + +and save. Now when you run `make test-unit` you will see our familiar +`pytest` output: + +```console + $ make test-unit +. [100%] +1 passed in 0.01s +``` + +Now, what other `make` test tasks might you want to define? Run `make +test` and you will see your options: + +```console + $ make test +. [100%] +1 passed in 0.01s +make test-lint not implemented: ./scripts/tests/lint.sh not found +make test-coverage not implemented: ./scripts/tests/coverage.sh not found +make test-contract not implemented: ./scripts/tests/contract.sh not found +make test-security not implemented: ./scripts/tests/security.sh not found +make test-ui not implemented: ./scripts/tests/ui.sh not found +make test-ui-performance not implemented: ./scripts/tests/ui-performance.sh not found +make test-integration not implemented: ./scripts/tests/integration.sh not found +make test-accessibility not implemented: ./scripts/tests/accessibility.sh not found +make test-capacity not implemented: ./scripts/tests/capacity.sh not found +``` + +You can see that the unit test we wrote has been run, and there are +several further test options predefined. They all follow the same +pattern as `scripts/tests/unit.sh`: by putting a shell script with the +right name under `scripts/tests`, you can control how the framework +executes that sort of test. + +Let's add another: let's get `make test-lint` working, and as an +arbitrary choice we'll use python's `black` tool. First add it to +`pyproject.toml` with `poetry`: + +```console +poetry add --group=dev black +``` + +Now, let's add it to the `test-lint` task. Edit a new file at +`scripts/tests/lint.sh` and add this: + +```shell +#!/bin/bash + +poetry run black --diff --check api/ +``` + +Save it and set it to be executable: + +```console +chmod +x scripts/tests/lint.sh +``` + +Now when I run `make test-lint` I see this: + +```console + $ make test-lint +--- /Users/alex/src/repository-template/api/test_hello_world.py 2023-09-13 15:31:39.692962+00:00 ++++ /Users/alex/src/repository-template/api/test_hello_world.py 2023-09-13 16:30:16.798188+00:00 +@@ -1,5 +1,6 @@ + from hello_world import lambda_handler ++ + + def test_lambda_handler(): + response = lambda_handler(None, None) + assert "message" in response +would reformat /Users/alex/src/repository-template/api/test_hello_world.py + +Oh no! 💥 💔 💥 +1 file would be reformatted, 1 file would be left unchanged. +make[1]: *** [scripts/tests/test.mk:68: _test] Error 1 +make: *** [scripts/tests/test.mk:15: test-lint] Error 2 +``` + +It's picked up that there's a blank line missing after our import. We +could fix that manually, but `black` can do it for us. Let's add a +`make` task we can run to format our code. Let's call it, +imaginatively, `make format`. + +Open the top-level `Makefile` in the project. You'll see some +predefined tasks that you can edit. There isn't one called `format` +yet, so we need to add it. + +Find the `sh` task, and in the space under it, add this `make` rule +definition: + +```make +format: # Apply code formatting + make _project name="format" + +``` + +Note, if you're not used to makefile syntax, that the space before +`make` on the second line needs to be a single `tab` character. It +will break otherwise. Copy and paste if in doubt. + +Close the `Makefile`, and create the file +`scripts/projecthooks/format.sh` with the contents: + +```shell +#!/bin/bash + +poetry run black api/ +``` + +Save it and run `chmod +x scrips/projecthooks/format.sh` to make it +executable. + +That's all you need to do: run `make format` and you will see +something like this: + +```console + $ make format +make _project name="format" +reformatted /Users/alex/src/repository-template/api/test_hello_world.py + +All done! ✨ 🍰 ✨ +1 file reformatted, 1 file left unchanged. +``` diff --git a/scripts/config/markdownlint.yaml b/scripts/config/markdownlint.yaml index 554ab554..1dcaf0b0 100644 --- a/scripts/config/markdownlint.yaml +++ b/scripts/config/markdownlint.yaml @@ -1,5 +1,11 @@ # SEE: https://github.com/DavidAnson/markdownlint/blob/main/schema/.markdownlint.yaml +# https://github.com/DavidAnson/markdownlint/blob/main/doc/md010.md +MD010: # no-hard-tabs + ignore_code_languages: + - make + - console + # https://github.com/DavidAnson/markdownlint/blob/main/doc/md013.md MD013: false From d990a7856980ed65136f81b78e61e30dfe09ede8 Mon Sep 17 00:00:00 2001 From: regularfry Date: Wed, 18 Oct 2023 17:09:09 +0100 Subject: [PATCH 2/6] Add back some removed Makefile niceties --- Makefile | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 4b497438..bbe79d73 100644 --- a/Makefile +++ b/Makefile @@ -3,12 +3,13 @@ include ./scripts/init.mk +# ============================================================================== + # Example CI/CD targets are: dependencies, build, publish, deploy, clean, etc. dependencies: # Install dependencies needed to build and test the project # TODO: Implement installation of your project dependencies - build: # Build the project artefact @Pipeline make _project name="build" @@ -34,6 +35,8 @@ config:: # Configure development environment (main) @Configuration # TODO: Use only 'make' targets that are specific to this project, e.g. you may not need to install Node.js make _install-dependencies +# ============================================================================== + _project: set -e SCRIPT="./scripts/projecthooks/${name}.sh" From e1d5b109c9bb730884cf9be1c0b4244754315ca2 Mon Sep 17 00:00:00 2001 From: regularfry Date: Thu, 19 Oct 2023 11:31:38 +0100 Subject: [PATCH 3/6] Update Tutorial.md to keep pace with `asdf` changes --- docs/{tutorial.md => Tutorial.md} | 70 +++++++++++-------------------- 1 file changed, 24 insertions(+), 46 deletions(-) rename docs/{tutorial.md => Tutorial.md} (84%) diff --git a/docs/tutorial.md b/docs/Tutorial.md similarity index 84% rename from docs/tutorial.md rename to docs/Tutorial.md index ee5f5edf..04e4d597 100644 --- a/docs/tutorial.md +++ b/docs/Tutorial.md @@ -1,10 +1,5 @@ # The Simplest Thing That Can Possibly Work -This repository template contains a lot of material, and provides a -lot of functionality out of the box. You might feel intimidated and -confused by the sheer number of files in it, and not know where to -look to get what you need done. - This tutorial is to walk you through the steps of getting a basic Hello-World-style app into production, to show you where to find the functionality you need and to demystify some of the content so that @@ -19,14 +14,13 @@ I will assume familiarity with git itself. ## First things first You will need certain tools to be installed in order to use the tools -in the repository template. They are: - -- The gnu coreutils. -- Gnu make. +in the repository template. They are listed in the +[`README`](../README.md#prerequisites) at the root of the project, so +make sure you have them installed first, before trying to continue. -These are some of the tools that the engineering community assumes you -will have for compatibility with a wide range of projects and -products, so install them if you haven't already. +These prerequisites are some of the tools that the engineering +community assumes you will have for compatibility with a wide range of +projects and products. Some other tools will be installed by the framework if you don't already have them installed. I'll call those out as and when they @@ -94,36 +88,20 @@ returns a JSON blob that looks like this: The framework contains a mechanism for ensuring that your local development python interpreter is at a known version; that mechanism -relies on [`asdf`](https://github.com/asdf-vm/asdf). The next step will -install it if you don't already have it, and will update it if you -do, but there's something we need to edit first. - -Open `Makefile` in your editor. This is the entrypoint for the tasks -that the framework supplies for you, and this entrypoint is something -you can edit for yourself. +relies on [`asdf`](https://github.com/asdf-vm/asdf). The next step +will use `asdf` to install dependencies for you, but there's something +we need to edit first. -Locate the `config` task. You will see that it looks like this: +The framework assumes that all tool dependencies and versions are +specified in the `.tool-versions` file, so open it and add these lines +at the top: -```make -config:: # Configure development environment - # TODO: Use only `make` targets that are specific to this project, e.g. you may not need to install Node.js - make \ - nodejs-install \ - python-install \ - terraform-install -``` - -For this project we don't need NodeJS, and while we will need -terraform, we don't need it yet. Modify the `config` task so that it -looks like this: - -```make -config:: # Configure development environment - make python-install +```text +python 3.11.4 +poetry 1.6.1 ``` -That means it will only install python for us, which - for now - is -the only thing we want. Run the task: +Run the task: ```shell make config @@ -131,10 +109,10 @@ make config The first time you run this it will download and build `python` for you, so you may want to make a cup of tea. The version it picks is in -the `.tool-version` file at the top level of the template. +the `.tool-versions` file at the top level of the template. -In addition to installing `asdf`, `make config` sets up some `git` -commit hooks that will come in handy later. +`make config` also sets up some `git` commit hooks that will come in +handy later. Now, the template has given us a `poetry` installation for dependency management. There is a configuration option that I recommend you set @@ -153,7 +131,7 @@ deleting the working directory and starting again will reset all (well, nearly all) the relevant state. However, you don't want to be committing the virtual environment to -`git`, so edit the file `.gitignore` and add : +`git`, so edit the file `.gitignore` and add: ```text .venv/ @@ -183,8 +161,8 @@ Now we add `pytest`: poetry add --group=dev pytest ``` -With that prologue out of the way, we can write our test. Which will -fail, but seeing it fail in the right way will tell us that the python +With that prologue out of the way, we can write our test. It will +fail, but seeing it fail in the right way will tell us that the Python environment is how we need it. First, make a directory to put our code into: @@ -203,7 +181,7 @@ def test_lambda_handler(): assert response["message"] == "Hello World" ``` -Run pytest with the command `poetry run pytest` and you will see the +Run `pytest` with the command `poetry run pytest` and you will see the following (among some other test failure info): ```console @@ -367,7 +345,7 @@ format: # Apply code formatting ``` -Note, if you're not used to makefile syntax, that the space before +Note, if you're not used to `make` syntax, that the space before `make` on the second line needs to be a single `tab` character. It will break otherwise. Copy and paste if in doubt. From f512881eb453d751cf67f8bc5b03084ccc3d46f3 Mon Sep 17 00:00:00 2001 From: regularfry Date: Thu, 19 Oct 2023 11:37:54 +0100 Subject: [PATCH 4/6] Remove a Sonarcloud TODO The TODO was unnecessary: the SonarCloud step in the build pipeline fails open if it's not configured. --- docs/Tutorial.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/docs/Tutorial.md b/docs/Tutorial.md index 04e4d597..76606f5d 100644 --- a/docs/Tutorial.md +++ b/docs/Tutorial.md @@ -61,10 +61,6 @@ make it private if you prefer. Under `Repository name`, give the repository a name of `nhset-hello-world`. -[TODO: This means the SonarCloud integration won't work, but we can't -get around that without being able to make repositories in a different -org.] - If a description will help you remember the purpose of this project in the future, put something meaningful to you in the `Description` field. From 1fa6e52c8835622551f91a8203d4da2dd3ead304 Mon Sep 17 00:00:00 2001 From: Dan Stefaniuk Date: Sat, 25 Nov 2023 08:14:43 +0000 Subject: [PATCH 5/6] Post rebase fixes that I missed when solving merge conflicts --- Makefile | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/Makefile b/Makefile index bbe79d73..47014052 100644 --- a/Makefile +++ b/Makefile @@ -1,25 +1,25 @@ # This file is for you! Edit it to implement your own hooks (make targets) into # the project as automated steps to be executed on locally and in the CD pipeline. -include ./scripts/init.mk +include scripts/init.mk # ============================================================================== # Example CI/CD targets are: dependencies, build, publish, deploy, clean, etc. -dependencies: # Install dependencies needed to build and test the project +dependencies: # Install dependencies needed to build and test the project @Pipeline # TODO: Implement installation of your project dependencies -build: # Build the project artefact @Pipeline +build: # Build the project artefact @Pipeline make _project name="build" -up: # Run your code +up: # Run your code @Pipeline make _project name="up" -down: # Stop your code +down: # Stop your code @Pipeline make _project name="down" -sh: up # Get a shell inside your running project, running it first if necessary +sh: up # Get a shell inside your running project, running it first if necessary @Development make _project name="sh" publish: # Publish the project artefact @Pipeline @@ -55,5 +55,4 @@ ${VERBOSE}.SILENT: \ deploy \ down \ sh \ - tmp/build_timestamp \ up \ From bd8bcfd081624bd5070909a4ab333437f289ab4f Mon Sep 17 00:00:00 2001 From: regularfry Date: Wed, 22 May 2024 10:55:42 +0100 Subject: [PATCH 6/6] Vale fixes --- docs/Tutorial.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/Tutorial.md b/docs/Tutorial.md index 76606f5d..21ebb19c 100644 --- a/docs/Tutorial.md +++ b/docs/Tutorial.md @@ -5,7 +5,7 @@ Hello-World-style app into production, to show you where to find the functionality you need and to demystify some of the content so that you can get the most out of it. -We'll be building a python AWS Lambda handler, so to complete the +We'll be building a Python AWS Lambda handler, so to complete the later parts of the tutorial you'll need AWS credentials that let you deploy to it. @@ -26,7 +26,7 @@ Some other tools will be installed by the framework if you don't already have them installed. I'll call those out as and when they come up. -I do not assume, dictate, or otherwite hint at the correct text +I do not assume, dictate, or otherwise hint at the correct text editor for you to be using. I just silently judge you for your choice. @@ -34,7 +34,7 @@ choice. The first step to your brand new app is to clone the repository. For the purposes of this tutorial, we're going to clone it into your -personal account rather than into any of the NHS github +personal account rather than into any of the NHS GitHub organisations. That's not what you'd be doing for anything we want to go into production, but it means you can keep it as a reference and we don't need to worry about naming collisions. @@ -43,14 +43,14 @@ Go to in your browser, and click the green button marked `Use this template`. From the dropdown that appears, select `Create a new repository`. -You will be taken to github's `Create a new repository` page. You -might notice that this is the same page as github shows you when you +You will be taken to GitHub's `Create a new repository` page. You +might notice that this is the same page as GitHub shows you when you create an ordinary repository in the web user interface, but pay attention to the `Repository template` field: you will see that it is pre-populated with the `nhs-england-tools/repository-template` value. That is as we want, so leave it as is. -Leave the `Include all branches` checkbox unticked. +Leave the `Include all branches` checkbox un-ticked. Under `Owner`, select yourself from the `Choose an owner` dropdown. Visibility options will appear under the Description field. The @@ -65,7 +65,7 @@ If a description will help you remember the purpose of this project in the future, put something meaningful to you in the `Description` field. -Click the green `Create repository` button, and github will show you a +Click the green `Create repository` button, and GitHub will show you a loading page while your new instance of the repository is created. Now that you have your repository, clone it to your local machine for @@ -83,7 +83,7 @@ returns a JSON blob that looks like this: ``` The framework contains a mechanism for ensuring that your local -development python interpreter is at a known version; that mechanism +development Python interpreter is at a known version; that mechanism relies on [`asdf`](https://github.com/asdf-vm/asdf). The next step will use `asdf` to install dependencies for you, but there's something we need to edit first. @@ -280,7 +280,7 @@ right name under `scripts/tests`, you can control how the framework executes that sort of test. Let's add another: let's get `make test-lint` working, and as an -arbitrary choice we'll use python's `black` tool. First add it to +arbitrary choice we'll use Python's `black` tool. First add it to `pyproject.toml` with `poetry`: ```console