Inconsistent diffs and behaviour

[jace-ys]

Problem Description

We've been using safe-settings to manage a bunch of our repos now and for the most part it's been great 🎉 Unfortunately, we've encountered a couple of bugs/inconsistencies while using it, which make it hard to be confident that the right settings are applied from an infrastructure-as-code perspective.

What is actually happening

A few things we've hit:

Inaccurate diffs

Occasionally we get inaccurate or unexpected diffs on the PR comments. For example:

This was for a change where we didn't modify the collaborators setting.

Doesn’t handle deletion of settings very well

We are trying to replace our existing branch protection rules with rulesets, but removing/emptying the branches setting didn't actually do a delete of the branch protection rules as we expected.

branches: []

rulesets:
- name: default
  target: branch
  enforcement: active
  ...

Another case is where we tried to remove the integration_id on a status check context, but it was not actually removed:

  - type: required_status_checks
    parameters:
      strict_required_status_checks_policy: true
      required_status_checks:
      - context: "my ci check"
        integration_id: 1234567 # Removing this line didn't actually do anything

Missing context on PR comments and status check results

We also noticed that the PR comments and status checks reported by safe-settings are not always very informative or have the right context. See the following examples:

Keen to know if these are genuine bugs or if there's something we've missed when setting up safe-settings! 🙏🏻

Context

Are you using the hosted instance of probot/settings or running your own?

We are running our own instance in a Kubernetes cluster

If running your own instance, are you using it with github.com or GitHub Enterprise?

We are using it with standard github.com (Team plan)

Version of probot/settings

Helm chart version 2.1.11 (appVersion 2.1.2)

Version of GitHub Enterprise

N/A