⚡📟 easly manage oncall for serverless services
Serverless services facilitate fast develop and deploy cycles. Once in production, these applcations should also have an answer for: "Okay, so how is this application supported when customers start using it"? The answer should not be to wait until customers tell you your service is down.
The Serverless framework leverages multiple providers that manage many operational aspects of the runtimes your application runs on but they do not manage the operational aspects of your application itself. That's up to you.
A number of great monitoring solutions exist and there are even serverless plugins to leverage them! Serverless oncall fills the gap of how you then act on these monitoring systems.
Serverless oncall leverages pagerduty*, a managed oncall scheduling, dispatch and notification hub, to map your application service to an oncall rotation that intended to support its operation. The intended target for this support ideally is the developers that own the service.
This plugin is intended to pair well with the AWS Alerts plugin but is not coupled to AWS serverless provider.
Install the plugin with npm
$ npm i -D serverless-oncallUpdate your serverless.yml file with the following
plugins:
- serverless-oncall
custom:
oncall:
# the escalation_policy you want pager duty alerts associated with
escalationPolicy: ${env:PD_ESC_POLICY,''}
# credential used to authenticate with pagerduty api
# visit https://{company}.pagerduty.com/api_keys to create one
apiKey: ${env:PD_API_KEY,''}
# a list of integrations to associate with your oncall service
# this will determine where monitoring systems send information to.
#
# currently only the "cloudwatch" and "transform" integration are supported but more integrations are
# planned for the future
# you may omit this configuration if you wish and configure these manually
# in the pagerduty web console
integrations:
- cloudwatch:
- tranform:
code: 'code here'This plugin currently requires a pagerduty api key to interact with the pagerduty api.
💡 You can learn how to get your own api key here
It is recommended to
not inline your api key's value directly in your serverless.yaml in plain text to avoid accidently checking it into source control.
You can leverage serverless variables as you would with other secret credentials to resolve this value without including it in your source code.
The example above is using env variables to resolve these values. You would then invoke a serverless command by providing them externally.
$ PD_ESC_POLICY=xxx PD_API_KEY=xxxxxxx npx serverless infoServerless framework provides many options for storing these externally from your source code. Pick one.
As mentioned above the way you intergrate a monitoring systems is with pagerduty. The delivery of this information to your oncall team is key so its worth considering your options
This integration is easy to configure but limited in the way information is delivered.
This integration requires some configuration but is more flexible with how information is delivered.
Below is an example appoach to keeping that configuration managable.
The transform integration lets you provde a snippet of javascript that pager duty will
run when it recieves an incident event. This lets you desired how you want your information
to be delievered and displayed to your oncall team. The example below leverages serverless frameworks ability externalize and dereference data in separate files.
custom:
oncall:
integrations:
# event transform integration (pagerduty's serverless event transformer)
- transform:
code: "${file(transformConfig.js):code}"The transform.code field is a string of javascript source. This assumes
transformConfig.js is a file that will provide that by loading in
an oncallHandler.js file.
$ cat transformConfig.js
module.exports.code = () => {
// read in javascript source
return require('fs').readFileSync('oncallHandler.js').toString();
}$ cat oncallHandler.js
// https://v2.developer.pagerduty.com/docs/creating-an-integration-inline
var helloWorld = {
event_type: PD.Trigger,
incident_key: PD.inputRequest.body.incident_id,
description: PD.inputRequest.body.description,
details: PD.inputRequest.body,
client: "Hello world",
client_url: "https://github.com/softprops/serverless-oncall"
};
PD.emitGenericEvents([helloWorld]);This approach also brings the potential of unit testing your transform code as it is now just a standalone javascript file. The pagerduty docs may help here.
serverless-oncall adds new commands to your servleress vocabulary. You can discover these new commands though the built-in --help option
$ npx serverless --help | grep oncallIn order to create or update an oncall service you will need to first select an appropriate escalation policy to associate the service with. You can think of roughly as selecting the target group of individuals to be notified. This plugin provides the escalationPolicies command for conveniene of listing these but does not manage them directly. Use the pagerduty UI do that instead.
$ npx serverless oncall escalationPoliciesYour organization will likely have more than one time. The list above will include escalations for
all teams. To limit this list to a single (your) team use the -t flag and provide that teams pagerduty team identifier.
$ npx serverless oncall escalationPolicies -t {team}This command will use the information described your serverless.yml oncall resource to create resources with a remote provider, currently pagerduty.
$ npx serverless oncall syncserverless-oncall integrates itself with the serverless built-in command, info and prints out high level information about the state of its associated oncall service. If you just want a quick link to the oncall service, use this.
$ npx serverless info* serverless-oncall may support other oncall providers in the future
- interface design. the current design relects an minimum viable design. this is subject to change.
- support more integrations
- support more oncall service providers
Contributions are welcome. Please read our contributing doc for more information.
Doug Tangren (softprops) 2018
