Service release checklist and minor readme improvements (#7)

Author: kibertoad

LICENSE

@@ -175,18 +175,7 @@
 
    END OF TERMS AND CONDITIONS
 
-   APPENDIX: How to apply the Apache License to your work.
-
-      To apply the Apache License to your work, attach the following
-      boilerplate notice, with the fields enclosed by brackets "[]"
-      replaced with your own identifying information. (Don't include
-      the brackets!)  The text should be enclosed in the appropriate
-      comment syntax for the file format. We also recommend that a
-      file or class name and description of purpose be included on the
-      same "printed page" as the copyright notice for easier
-      identification within third-party archives.
-
-   Copyright [yyyy] [name of copyright owner]
+   Copyright 2023 Lokalise
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

README.md

@@ -6,12 +6,12 @@
 
 It comes with the following out-of-the-box:
 
-* fastify-based general application skeleton;
+* [fastify](https://www.fastify.io/docs/latest/) as a basis for the general web application skeleton;
 * Modular, domain-driven structure that encourages separation of concerns;
 * Server/app separation, for convenient bootstrapping in e2e tests;
 * [Global error handler](./src/infrastructure/errors/errorHandler.ts);
 * JSON-based, single line standardized [logging](./src/infrastructure/logger.ts);
-* Populates `req.id` for incoming requests based on `x-request-id` header, or generates new UUID if none is set.
+* Automatic population of `req.id` for incoming requests based on `x-request-id` header, or generation of new UUID if none is set, for the purposes of distributed tracing.
 
 Mechanisms:
 
@@ -102,8 +102,8 @@ npm run db:update-client
 docker compose up -d
 ```
 
-6. To register local dev requests in newrelic please use:
+6. To run application:
 
 ```shell
-NEW_RELIC_APP_NAME=dev.yourapp.yourdomain.com NEW_RELIC_LICENSE_KEY=<license_key> npm run start:dev
+npm run start:dev
 ```

docs/service-release-checklist.md

@@ -0,0 +1,104 @@
+# Service readiness checklist
+
+This document aims to provide a checklist for determining whether service is mature enough for general availability.
+
+## Checklist
+
+Legend: `M` - Mandatory, `R` - Recommended
+
+### Documentation
+
+- `M`: README.md in the repository:
+  * service description, ownership, links to documentation,
+    build and deployment instructions, configuration variables, healthchecks, etc
+- `R`: Architectural diagram for the service
+- `R`: SLI/SLO/SLA definitions (at least service criticality level: P1/2/3)
+
+### Build and release pipeline
+
+- `M`: GitHub Actions worklows for building the code, running linting check and executing tests
+- `M`: Artifacts are container images
+
+### Infrastructure
+
+- `M`: All persistent state (if any) is stored in external storage
+- `M`: All components must be designed for HA (e.g. an app should be able to run as
+  multiple instances in active/active configuration)
+- `M`: Deployment diagram
+- `M`:
+  - traffic types (HTTP/gRPC/other)
+  - spikes / static IP address
+  - adv L7 feat.: waf/auth/sticky sessions/request routing/load balancing algs
+  - CORS requirements
+
+### Security
+
+- `M`: No secrets in the code
+- `M`: Any housekeeping/one-off/migration/etc tasks must be part of the
+  application; `stage` and `live` environment are not accessible directly.
+- `M`: Externally exposed services must require authentication
+- `M`: Documentation must have answers for the following questions:
+  * Is this internal or external service?
+  * Does the service make any outbound connections? If yes, specify destinations.
+  * Does the service handle personally identifiable information?
+- `R`: HTTP headers / CORS:
+  * `X-Frame-Options`, `Strict-Transport-Security`, `X-XSS-Protection`,
+    `X-DNS-Prefetch-Control`
+
+### Operations
+
+- `M`: Application configuration is set via environment variables
+- `M`: Logging satisfies the following requirements:
+  * single-line json to stdout/stderr, `message` or `msg` field at the root level
+  * make sure data types for json fields aren't mixed otherwise parsing will not work
+  * at least two verbosity levels: debug/error;
+    * `error`: unexpected error that prevents further processing
+    * `warn` : irregular events with defined recovery strategy
+    * `info` : major state changes; must log: component start, became operational,
+      event/task processed, shutdown started, just before exited
+    * `debug`: diagnostic and troubleshooting event
+  * global error handler; make sure all errors are logged
+  * error response structure adheres to defined standard
+  * field `level` must contain a string (error, warn) and not a number
+  * distributed tracing:
+    * request id is passed via `x-request-id` header, must propagate if received, otherwise generate a new one
+    * request id must be included with the request-scoped logging and outgoing HTTP requests
+- `M`: Implements APM integration
+- `M`: Healthcheck endpoint; should provide:
+  * at a minimum: 200 response if the service is operational, non-200 response code otherwise
+  * include app version and commit hash in the response
+  * recommended: [readiness and liveness endpoints]
+- `M`: Implements metrics:
+  * 4 golden signals: latency/traffic/errors/saturation
+  * endpoint (preferably `/metrics`) in Prometheus format on a separate port (eg `9090`)
+  * availability, authentication status, and latency for all backend services
+  * Node.js metrics
+  * business metrics as necessary/defined by the service owner
+- `R`: Perform simple load testing of the service, use the results for:
+  * sizing the live infrastructure; eg cores, RAM, storage size
+  * define alerting thresholds; eg: 4 golden signals, latency/traffic(req cnt)/error/saturation
+
+### Resiliency
+
+- `M`: The service can run in multiple instances simultaneously
+- `M`: Must handle component unavailability gracefully (eg. unable to connect to storage):
+  - all connections must have reasonable timeouts and error handling
+  - all HTTP calls must have reasonable timeouts and error handling
+  - reconnect with exponential back-off where necessary
+- `M`: [Graceful shutdown] on SIGTERM (15): stop accepting connections, complete in-flight work, exit
+
+## References
+
+- https://www.opslevel.com/blog/production-readiness-in-depth#deployment
+- https://gruntwork.io/devops-checklist/
+- https://aleksei-kornev.medium.com/production-readiness-checklist-for-backend-applications-8d2b0c57ccec
+- https://github.com/mercari/production-readiness-checklist/blob/master/docs/references/pre-production-checklist.md
+- https://blog.last9.io/deployment-readiness-checklists/
+- https://habr.com/en/post/438186/
+- https://12factor.net
+- https://cloud.google.com/blog/products/containers-kubernetes/your-guide-kubernetes-best-practices
+
+[readiness and liveness endpoints]: https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-setting-up-health-checks-with-readiness-and-liveness-probes
+[Graceful shutdown]: https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-terminating-with-grace
+
+This document is based on a Lokalise Service Release Checklist, prepared by the Lokalise Platform Squad.