Proposal: Enhanced GitHub Action Experience

[hanna-skryl]

Following up on #149

Based on the research findings, this proposal outlines the implementation approach for GitHub App authentication to achieve branded comments.

Problem

Code PushUp users receive generic github-actions[bot] comments that don't reflect our professional identity. This limits the brand's visibility and credibility in the developer ecosystem.

Solution

GitHub App installation authentication with an optional marketplace listing

Why GitHub App installation authentication?

• Replaces generic github-actions[bot] with code-pushup-staging[bot] (or code-pushup[bot]) for clear brand attribution
• Provides more precise permission control
• Lays the foundation for future advanced integrations and features
• Is an industry standard; many successful tools consistently require GitHub Apps installation
• Requires zero breaking changes

User experience flow

1. Without app: User gets standard github-actions[bot] comments (current behavior)
2. With app: User gets branded code-pushup-staging[bot] comments
3. Automatic detection: No user configuration needed after app installation

Implementation details

Users must install the Code PushUp GitHub App on their repository or organization to receive branded comments. The GitHub Action cannot authenticate with an app that's only installed on the Code PushUp organization.

Technical requirements

• GitHub App ID and private key stored as repository secrets
• Permissions: pull-requests: write (for posting comments)
• The user must grant app access to target repositories during installation

Authentication process

1. User installs the Code PushUp GitHub App on their repository/organization
2. Action detects an app installation on the current repository during workflow execution
3. Action uses the @octokit/auth-app library to authenticate as the GitHub App and generate installation tokens
4. Action uses an installation token for API calls instead of the default github.token
5. Comments appear as an app identity instead of a generic bot
6. Graceful fallback to the standard token when the app is not installed

Implementation scope

Proof of Concept

• An authentication system that can detect when the Code PushUp GitHub App is installed on a user's repository
• Token generation logic that creates GitHub App installation tokens for API calls
• Fallback mechanism that uses a standard GitHub token when the app is not installed
• Professional GitHub App page with a meaningful description (the current app has no user-facing documentation)
• Documentation updates increasing app visibility (currently, no repositories mention the app)

Discussion points

• App identity: Should we use code-pushup-staging or create a more general code-pushup app?
• Permission compatibility: Does the current app have sufficient permissions?
• Marketplace priority: Should we create a GitHub Marketplace listing to help users discover the app (requirements for listing an app)?

[matejchalk]

Thanks for the write-up, the proposal sounds good to me 👍 Fallback to default github.token also makes it a non-breaking change, which is welcome.

App identity: Should we use code-pushup-staging or create a more general code-pushup app?

I'd create a separate code-pushup app for production. We can use the code-pushup-staging app for testing this feature in our own repos, but as the name suggests, it's not intended for users outside our organization.

• Permission compatibility: Does the current app have sufficient permissions?

I believe not. It's currently only used to give portal access to GitHub API, so the permissions are restricted for this purpose. Specifically, it only has Read-only Metadata (mandatory) and Read-only Contents (selected) permissions.

So I believe we'd need to add a Read and write Pull requests permission, is that right?

• Marketplace priority: Should we create a GitHub Marketplace listing to help users discover the app (requirements for listing an app)?

Not sure about this. I think we can list it as a free app, but even so it seems like there are some hoops to jump through and I'm not sure it gives us much advantage to be discoverable through the GitHub Marketplace. What do you think, @vmasek?

[vmasek]

I would not prioritize creating market place app for now, but having separate app for prod and using our logo for the icon would definitely be a nice touch

[hanna-skryl]

Makes perfect sense to set up a production app. I can create a ticket for someone with owner/admin privileges to do this.

So I believe we'd need to add a Read and write Pull requests permission, is that right?

I think I was too quick to add pull-requests: write permission in the proposal technical requirements. Looking at the code, the action uses GitHub's Issues API endpoints to manage PR comments. So, the app will likely need Issues: Read and write permissions. But this needs to be tested.

I would not prioritize creating market place app for now

Agreed. A marketplace listing can always be added later if we find users have trouble discovering the app.

I'm thinking we'll need two separate tickets:

1. GitHub Apps setups:
   • configure code-pushup-staging app with proper permissions
   • create production code-pushup app
   • add both apps' IDs and private keys to repository secrets
2. Implementation. Once the apps are ready, someone can implement the authentication changes using the PoC from the proposal

One question, though: how do you think we should approach testing this feature on Code PushUp repositories before creating a public release?

[matejchalk]

One question, though: how do you think we should approach testing this feature on Code PushUp repositories before creating a public release?

This is suprisingly easy when it comes to GitHub Actions, we can reference any Git branch, commit or tag in our workflows. For example, let's say you had a work-in-progress implementation pushed to a github-app-auth branch. Then you can create a dummy PR in code-pushup/cli with the following change and it'll execute your code:

diff --git a/.github/workflows/code-pushup.yml b/.github/workflows/code-pushup.yml
index cc135147..c8be8069 100644
--- a/.github/workflows/code-pushup.yml
+++ b/.github/workflows/code-pushup.yml
@@ -34,6 +34,6 @@ jobs:
       - name: Install dependencies
         run: npm ci
       - name: Run Code PushUp action
-        uses: code-pushup/github-action@v0
+        uses: code-pushup/github-action@github-app-auth
         with:
           bin: npx nx code-pushup --

[matejchalk]

Looks like adding permissions will prompt existing GitHub users to accept them 🤔

So maybe it would best to test it with a new GitHub App until it's ready. I can create it, just ping me when you need it. I'm guessing you'll want to implement it, @hanna-skryl?

[hanna-skryl]

@matejchalk Sure, I'd like to try and implement it. There is no rush on the app setup. If you can do it this week, that's great, but it can wait until later.