Eng Pipes

This repo contains automation infrastructure for the engineering organization, including:

• CI tooling used in sentry and getsentry development
• metrics collection and aggregation (for e.g. GitHub)
• a Slack integration to provide tools through Slack

Read the README in the src/ directory for information on the code structure.

Architecture

The services that this app is configured for, send metrics to a web app. This app stores the data in Google's BigQuery. A Looker dashboard (here's an example) shows the data it fetches from BigQuery.

The app can also determine that it needs to send a message to the Slack workspace.

Sentry bot Slack app

The Sentry bot Slack app sends events to the production backend. You can find what URL are events sent to by going to "Event subscriptions" in the app page. Events are sent to the "apps/slack/events" route.

The events sent by the bot are defined under "Subscribe to bot events" in the "Event subscriptions" page. The current set of events (which can change over time) are:

• User clicked into your App Home
• Subscribe to only the message events that mention your app or bot
• A message was posted in a direct message channel
• A member's data has changed.

GitHub Sentry Webhooks

Under Sentry's webhooks there's webhooks to the production backend with the route "webhooks/github".

Pre-requisites

• direnv - Recommended to load env variables
• Docker
• Yarn

Setup

direnv will create a .env file for you if you don't have one. Follow the instructions below and adjust the variables in the .env file.

Setup Secrets

The following secrets are configured in GitHub for this app to function and to deploy to Google. You can grab GitHub secrets in their respective configuration pages: GitHub App

Local Secrets (required to run yarn dev)

You will also need to set up some of these environment variables if you want to test this locally, e.g. using direnv or something similar

• GH_APP_PRIVATE_KEY - GitHub App private key for your test app. It needs to all be on one line, with literal \ns instead of newlines (these seem to be required).
• GH_WEBHOOK_SECRET - GitHub webhook secret to confirm that webhooks come from GitHub
• SENTRY_WEBPACK_WEBHOOK_SECRET - Webhook secret that needs to match secret from CI. Records webpack asset sizes.
• SLACK_SIGNING_SECRET - Slack webhook secret to verify payload that you can get from Basic Information -> App Credentials
• SLACK_BOT_USER_ACCESS_TOKEN - The Slack Bot User OAuth Access Token from the Oauth & Permissions section of your Slack app

After creating your github org, make sure to also add its slug to .env file:

• GETSENTRY_ORG_SLUG - Github slug of the org you create in Configuring a test environment below
• SENTRY_REPO - Github slug of your test repo that you create in Configuring a test environment below

Optional database configuration

• DB_USER - Database user
• DB_PASSWORD - Database password
• DB_NAME - Database name
• DB_INSTANCE_CONNECTION_NAME - Used for CloudSQL

Production Secrets

These environment vars are used for deploying to GCP

• GOOGLE_SERVICE_ACCOUNT_EMAIL - Google service account email
• GOOGLE_APPLICATION_CREDENTIALS - Google service account private key

And finally, for Sentry releases

• SENTRY_AUTH_TOKEN - Auth token used to create releases
• SENTRY_PROJECT - the Sentry project id

Google Cloud Platform

You'll need to setup a GCP project that has access to Google Cloud Run. You should create a service account that has the following roles:

• Cloud Run Admin
• Service Account User
• Cloud SQL Client

You'll also need to create a private key for the service account (it should download a JSON file). You'll want to run base64 <path/to/json> and set it as the GOOGLE_APPLICATION_CREDENTIALS secret in GitHub.

Configuring a test environment

1. Set up Ngrok to redirect calls to your localhost (smee.io also works).

   • If you haven't already, sign up for an ngrok account and grab the auth token from the setup page.
   • ngrok config add-authtoken <YOUR_NGROK_AUTH_TOKEN>
   • ngrok http 3000 --> Grab the URL ngrok gives you (e.g. https://6a88fe29c5cc.ngrok.io henceforth referred to as NGROK_INSTANCE) and save it for step 6

2. Create a new personal Slack workspace from the Slack app (e.g. Sentry (testing)). Do not use the Sentry workspace!

   • This workspace should be using your @sentry.io account otherwise you'll have a bunch of issues due to the built-in @sentry.io checks in this app.

3. Create a new personal Slack App that matches the settings of the production app

   • The prompt will ask you to associate to a workspace (use the new workspace you made in step 2)

4. In order for your Slack app to work, you need to match the settings to the production Slack app. Run the following command and copy the output into the slack app's App Manifest, making sure to use https:// URLs instead of http:// ones.

   sed 's|<NGROK_URL>|https://<NGROK_INSTANCE>.ngrok.io|g' ./.slack-manifest.example

   If you have jq installed and wish to avoid some manual copy-pasting, the following mini-script will also achieve the same effect once ngrok http is running in another terminal:

   sed "s|<NGROK_URL>|$( curl -s localhost:4040/api/tunnels | jq -r '.tunnels[0].public_url')|g" ./.slack-manifest.example | pbcopy

   • Some of the settings will need to be verified before they get saved
   • This means that you will need to update your .env file with the settings from your Slack app
   • Reload your server for the new env vars to apply and resend the verification payloads
   • You will have to do this with multiple settings, thus, you will have to repeat reloading your server as you add new variables

5. Create a GitHub organization. Name it what you like.

   • After the organization has been created, go to its Settings, then select Personal access tokens from the sidebar to enable tokens.
   • In the setup menu, select Allow access via fine-grained personal access tokens, then Do not require administrator approval, and finally Allow access via personal access tokens (classic).
   • In your organization, create a new repository named something like testing-eng-pipes.

6. Create a personal access token.

   • Title the new token Eng-pipes development token, give this token 90 days until expiration, and enable the following permissions: read:org and read:user.

   • On the next page, copy the displayed token into the GH_USER_TOKEN field of your .env file.

     ⚠️ You are giving this token permissions to all orgs across GitHub that you are a member of (though some, like getsentry, are configured to require approval before PATs have access). Be careful and ensure it does not leave your machine!

7. Create a GitHub App.

   • Set the webhook to your ngrok tunnel with the GH route (e.g. <NGROK_INSTANCE>/webhooks/github)
   • Create and download a private key and add it to your .env under GH_APP_PRIVATE_KEY. You'll need to convert newlines to literal \n. (See also Setup Secrets above.)
   • Go to the Permissions & events sidebar menu entry of the GitHub app configuration, and grant maximum non-Admin access (Read and write where possible, Read only everywhere else) for every line in Repository permissions (NOTE: We use a more constrained permission-set in production, but for initial setup enabling maximal permissions is fine; permissions can be whittled down later as needed.)
   • For Organization permissions, grant Read and write for Members and Projects
   • In the Subscribe to events section, check every possible box
   • Go to Install App in the sidebar menu of the GitHub app configuration, and install the app for your GitHub organization.
   • When prompted, choose All repositories.

8. In your GitHub organization, create a new project called GitHub Issues Someone Else Cares About.

   • Go to the project's Settings, then modify the Status field to only have the following options (note the capitalization): Waiting for: Community, Waiting for: Product Owner, and Waiting for: Support.
   • Add a new field (note the capitalization) called Response Due of type Text.
   • Add a new field (note the capitalization) called Product Area of type Single select, with the following options: Alerts, Crons, Dashboards, Discover, Issues, Performance, Profiling, Projects, Relays, Releases, and User Feedback.

9. In your GitHub repository at [your-org]/testing-eng-pipes-or-whatever, go to Issues, then click Labels.

   • Add the labels Waiting for: Community, Waiting for: Product Owner, and Waiting for: Support,
   • Add the labels Product Area: Alerts, Product Area: Crons, Product Area: Dashboards, Product Area: Discover, Product Area: Issues, Product Area: Performance, Product Area: Profiling, Product Area: Projects, Product Area: Relays, Product Area: Releases, and Product Area: User Feedback

10. Copy the file github-orgs.example.yml to github-orgs.local.yml.

    • Set an environment variable, GH_ORGS_YML=github-orgs.local.yml.

    • Modify the top-level key to the slug of your organization.

    • Set appId to the "App ID" from General > About in the UI for your app.

    • Set installationId to the ID at the end of the URL for your app's installation on your org (confused yet?).

    • Leave the privateKey as-is, it's the name of an environment variable to pull from (the main github-orgs.yml holds public config and is checked into version control).

    • In a terminal, log into the GitHub CLI using gh auth login.

    • Use the script at bin/get-project-ids.sh $orgSlug $projectNumber to determine the ids to set in github-orgs.yml for your project.

11. Follow the steps of the "Development & tests" section below to get the server running.

    • It will fail if you don't have all the correct env variables defined

12. Verify that the Slack -> eng-pipes server pipeline works

    • On your new Slack workspace, send a message to the bot
    • You should see your localhost app respond with a 200 status code:

    

13. Verify that the GitHub -> eng-pipes server pipeline works

    • In your testing-eng-pipes GitHub repository, create a new issue
    • You should see your localhost app response with a 200 status code:

    

14. Verify that the GitHub -> eng-pipes server -> Slack pipeline works

    • In your Slack workspace, create a #test channel, and in that channel type /notify-for-triage Alerts sfo to setup Slack messages
    • Create a new issue in [your-org]/testing-eng-pipes, then add the Product Area: Alerts label to it
    • You should see a 200 status code in the ngrok window, plus a description of the message in the yarn dev window, and a Slack message describing the new issue in the #test channel of your Slack workspace

Congratulations, you're all setup!

Quality-of-life tips

• ngrok gives you a localhost interface to see events coming and to replay them.

• If you have an ngrok Pro account, you can define your own domain, saving you a round trip of having to update the Slack manifest and restarting your server to ingest the autogenerated ngrok domain. You can do this by setting the --domain flag when you invoke like so:

ngrok http 3000 --domain your-fun-unique-subdomain.ngrok.io

• GitHub lets you see web hooks events it recently delivered and even redeliver them if needed. Simply go to the Advanced section of your GitHub app's settings page, select the event of interest from Recent Deliveries, and click the Redeliver button to send it again.

• If BigQuery is producing a bunch of logspam for you, try adding DRY_RUN=true or DRY_RUN=1 to your .env file.

• If you open this project in VSCode, you can attach the built-in debugger to a running server. Rather than using yarn dev from the command line, instead open VSCode and select Debug: Start Debugging from the command palette:

You should hit any breakpoints you set in the code:

Development & tests

Follow the steps in the '''Setup''' section before running these steps:

# Start local postgres
docker run --rm --name ci-tooling-postgres -e POSTGRES_PASSWORD=docker -d -p 127.0.0.1:5434:5432 postgres:12
# Install dependencies
yarn install
# Update DB
yarn migrate:latest
# Start dev (it won't work until you set up some of the variables in .env file)
yarn dev

Running tests:

# Testing
yarn test

Troubleshooting

If after installing dependencies via yarn install you see 'Cannot find module' errors, make sure you are using the workspace's version of typescript. You should see a prompt in the bottom right of VSCode asking you to install the workspace version of typescript. If you don't see this prompt, you can set it manually by opening the command palette by pressing cmd+shift+p and selecting TypeScript: Select TypeScript Version. Then select Use Workspace Version.

Setting up new projects

This section only matters if you want to gather metrics from other projects than the ones we currently do.

Install the GitHub application to relevant repos (you will need to contact IT for access to this app). This app is used for GitHub API access to the repos it is installed on.

Deploying

All pushes to main will deploy to the existing GCP project.

TODO

Add notes for the following:

• What code does a service need to call to report to this app?
• How do you build a Looker dashboard?