Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wire in GitHub App authentication for the commenter and update job flags #32806

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,8 @@ periodics:
-project:kubernetes/169
NOT "complete the pre-review checklist and request an API review"
- --updated=5m
- --token=/etc/github-token/token
- --endpoint=http://ghproxy.default.svc.cluster.local
- --github-token=/etc/github-token/token
- --github-endpoint=http://ghproxy.default.svc.cluster.local
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this isn't a sufficiently safe way to make this change, as we're not running from source here, see the image above

- |-
--comment=This PR [may require API review](https://git.k8s.io/community/sig-architecture/api-review-process.md#what-apis-need-to-be-reviewed).

Expand Down Expand Up @@ -71,8 +71,8 @@ periodics:
label:area/stable-metrics
NOT "documentation for the requirements and lifecycle of stable metrics"
- --updated=5m
- --token=/etc/github-token/token
- --endpoint=http://ghproxy.default.svc.cluster.local
- --github-token=/etc/github-token/token
- --github-endpoint=http://ghproxy.default.svc.cluster.local
- |-
--comment=This PR [may require stable metrics review](https://git.k8s.io/community/contributors/devel/sig-instrumentation/metric-stability.md).

Expand Down Expand Up @@ -111,8 +111,8 @@ periodics:
is:open
-label:"cncf-cla: no"
-label:"cncf-cla: yes"
- --token=/etc/github-token/token
- --endpoint=http://ghproxy.default.svc.cluster.local
- --github-token=/etc/github-token/token
- --github-endpoint=http://ghproxy.default.svc.cluster.local
- |-
--comment=Unknown CLA label state. Rechecking for CLA labels.

Expand Down Expand Up @@ -161,8 +161,8 @@ periodics:
-label:priority/critical-urgent,priority/important-soon,priority/important-longterm
label:lifecycle/rotten
- --updated=720h
- --token=/etc/github-token/token
- --endpoint=http://ghproxy.default.svc.cluster.local
- --github-token=/etc/github-token/token
- --github-endpoint=http://ghproxy.default.svc.cluster.local
- |-
--comment=The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

Expand Down Expand Up @@ -218,8 +218,8 @@ periodics:
-label:lifecycle/frozen
label:lifecycle/rotten
- --updated=720h
- --token=/etc/github-token/token
- --endpoint=http://ghproxy.default.svc.cluster.local
- --github-token=/etc/github-token/token
- --github-endpoint=http://ghproxy.default.svc.cluster.local
- |-
--comment=The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

Expand Down Expand Up @@ -286,8 +286,8 @@ periodics:
repo:kubernetes/kops
repo:kubernetes/kubernetes
repo:kubernetes/test-infra
- --token=/etc/github-token/token
- --endpoint=http://ghproxy.default.svc.cluster.local
- --github-token=/etc/github-token/token
- --github-endpoint=http://ghproxy.default.svc.cluster.local
- |-
--comment=The Kubernetes project has merge-blocking tests that are currently too flaky to consistently pass.

Expand Down Expand Up @@ -347,8 +347,8 @@ periodics:
-label:"triage/accepted"
label:lifecycle/stale
- --updated=720h
- --token=/etc/github-token/token
- --endpoint=http://ghproxy.default.svc.cluster.local
- --github-token=/etc/github-token/token
- --github-endpoint=http://ghproxy.default.svc.cluster.local
- |-
--comment=The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

Expand Down Expand Up @@ -405,8 +405,8 @@ periodics:
-label:lifecycle/rotten
label:lifecycle/stale
- --updated=720h
- --token=/etc/github-token/token
- --endpoint=http://ghproxy.default.svc.cluster.local
- --github-token=/etc/github-token/token
- --github-endpoint=http://ghproxy.default.svc.cluster.local
- |-
--comment=The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

Expand Down Expand Up @@ -466,8 +466,8 @@ periodics:
-label:"good first issue"
-label:"triage/accepted"
- --updated=2160h
- --token=/etc/github-token/token
- --endpoint=http://ghproxy.default.svc.cluster.local
- --github-token=/etc/github-token/token
- --github-endpoint=http://ghproxy.default.svc.cluster.local
- |-
--comment=The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

Expand Down Expand Up @@ -524,8 +524,8 @@ periodics:
-label:lifecycle/stale
-label:lifecycle/rotten
- --updated=2160h
- --token=/etc/github-token/token
- --endpoint=http://ghproxy.default.svc.cluster.local
- --github-token=/etc/github-token/token
- --github-endpoint=http://ghproxy.default.svc.cluster.local
- |-
--comment=The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

Expand Down Expand Up @@ -576,8 +576,8 @@ periodics:
org:kubernetes-csi
label:lifecycle/frozen
is:pr
- --token=/etc/github-token/token
- --endpoint=http://ghproxy.default.svc.cluster.local
- --github-token=/etc/github-token/token
- --github-endpoint=http://ghproxy.default.svc.cluster.local
- |-
--comment=The `lifecycle/frozen` label can not be applied to PRs.

Expand Down Expand Up @@ -629,8 +629,8 @@ periodics:
-label:priority/critical-urgent
is:issue
- --updated=8760h
- --token=/etc/github-token/token
- --endpoint=http://ghproxy.default.svc.cluster.local
- --github-token=/etc/github-token/token
- --github-endpoint=http://ghproxy.default.svc.cluster.local
- |-
--comment=This issue has not been updated in over 1 year, and should be re-triaged.

Expand Down Expand Up @@ -677,8 +677,8 @@ periodics:
label:priority/important-soon
is:issue
- --updated=2160h
- --token=/etc/github-token/token
- --endpoint=http://ghproxy.default.svc.cluster.local
- --github-token=/etc/github-token/token
- --github-endpoint=http://ghproxy.default.svc.cluster.local
- |-
--comment=This issue is labeled with `priority/important-soon` but has not been updated in over 90 days, and should be re-triaged.
Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Expand Down Expand Up @@ -727,8 +727,8 @@ periodics:
label:priority/critical-urgent
is:issue
- --updated=720h
- --token=/etc/github-token/token
- --endpoint=http://ghproxy.default.svc.cluster.local
- --github-token=/etc/github-token/token
- --github-endpoint=http://ghproxy.default.svc.cluster.local
- |-
--comment=This issue is labeled with `priority/critical-urgent` but has not been updated in over 30 days, and should be re-triaged.
Critical-urgent issues must be actively worked on as someone's top priority right now.
Expand Down
56 changes: 22 additions & 34 deletions robots/commenter/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -29,14 +29,12 @@ import (
"fmt"
"log"
"math/rand"
"net/url"
"regexp"
"strconv"
"strings"
"text/template"
"time"

"sigs.k8s.io/prow/pkg/config/secret"
"sigs.k8s.io/prow/pkg/flagutil"
"sigs.k8s.io/prow/pkg/github"
)
Expand All @@ -58,9 +56,9 @@ const (
)

func flagOptions() options {
o := options{
endpoint: flagutil.NewStrings(github.DefaultAPIEndpoint),
}
o := options{}

flag.StringVar(&o.org, "org", "", "GitHub organization (required when using GitHub App credentials)")
flag.StringVar(&o.query, "query", "", "See https://help.github.com/articles/searching-issues-and-pull-requests/")
flag.DurationVar(&o.updated, "updated", 2*time.Hour, "Filter to issues unmodified for at least this long if set")
flag.BoolVar(&o.includeArchived, "include-archived", false, "Match archived issues if set")
Expand All @@ -70,10 +68,10 @@ func flagOptions() options {
flag.StringVar(&o.comment, "comment", "", "Append the following comment to matching issues")
flag.BoolVar(&o.useTemplate, "template", false, templateHelp)
flag.IntVar(&o.ceiling, "ceiling", 3, "Maximum number of issues to modify, 0 for infinite")
flag.Var(&o.endpoint, "endpoint", "GitHub's API endpoint")
flag.StringVar(&o.graphqlEndpoint, "graphql-endpoint", github.DefaultGraphQLEndpoint, "GitHub's GraphQL API Endpoint")
flag.StringVar(&o.token, "token", "", "Path to github token")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we keep these temporarily and backfill o.github.* items from them to avoid breaking existing users (outside of k8s)?

I'd prefer to do that and emit a warning about eventually (a month? two?) removing these options, to give folks at least a chance to notice and fixup their configs. With a reminder comment someone will eventually notice and remove the code after the date.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(as written this will also break kubernetes, unless quickly followed with a successful image upgrade ...)

Copy link
Contributor Author

@tnozicka tnozicka Jun 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've started by wiring the old flags, but some of the fields were private (in the new prow utils config). Let me give it a second try with some extra variables and syncing it back.

flag.BoolVar(&o.random, "random", false, "Choose random issues to comment on from the query")

o.github.AddFlags(flag.CommandLine)

flag.Parse()
return o
}
Expand All @@ -88,17 +86,16 @@ type meta struct {
type options struct {
ceiling int
comment string
org string
includeArchived bool
includeClosed bool
includeLocked bool
useTemplate bool
query string
endpoint flagutil.Strings
graphqlEndpoint string
token string
updated time.Duration
confirm bool
random bool
github flagutil.GitHubOptions
}

func parseHTMLURL(url string) (string, string, int, error) {
Expand Down Expand Up @@ -151,8 +148,8 @@ func makeQuery(query string, includeArchived, includeClosed, includeLocked bool,
}

type client interface {
CreateComment(owner, repo string, number int, comment string) error
FindIssues(query, sort string, asc bool) ([]github.Issue, error)
CreateComment(org, repo string, number int, comment string) error
FindIssuesWithOrg(org, query, sort string, asc bool) ([]github.Issue, error)
}

func main() {
Expand All @@ -162,31 +159,22 @@ func main() {
if o.query == "" {
log.Fatal("empty --query")
}
if o.token == "" {
log.Fatal("empty --token")
if o.github.TokenPath == "" && o.github.AppID == "" {
log.Fatal("no github authentication options specified")
}
if o.github.AppID != "" && o.org == "" {
log.Fatal("using github appid requires using --org flag")
}
if o.comment == "" {
log.Fatal("empty --comment")
}

if err := secret.Add(o.token); err != nil {
log.Fatalf("Error starting secrets agent: %v", err)
githubOptsErr := o.github.Validate(true)
if githubOptsErr != nil {
log.Fatalf("Error validating github options: %v", githubOptsErr)
}

var err error
for _, ep := range o.endpoint.Strings() {
_, err = url.ParseRequestURI(ep)
if err != nil {
log.Fatalf("Invalid --endpoint URL %q: %v.", ep, err)
}
}

var c client
if o.confirm {
c, err = github.NewClient(secret.GetTokenGenerator(o.token), secret.Censor, o.graphqlEndpoint, o.endpoint.Strings()...)
} else {
c, err = github.NewDryRunClient(secret.GetTokenGenerator(o.token), secret.Censor, o.graphqlEndpoint, o.endpoint.Strings()...)
}
c, err := o.github.GitHubClient(!o.confirm)
if err != nil {
log.Fatalf("Failed to construct GitHub client: %v", err)
}
Expand All @@ -202,7 +190,7 @@ func main() {
asc = true
}
commenter := makeCommenter(o.comment, o.useTemplate)
if err := run(c, query, sort, asc, o.random, commenter, o.ceiling); err != nil {
if err := run(c, o.org, query, sort, asc, o.random, commenter, o.ceiling); err != nil {
log.Fatalf("Failed run: %v", err)
}
}
Expand All @@ -221,9 +209,9 @@ func makeCommenter(comment string, useTemplate bool) func(meta) (string, error)
}
}

func run(c client, query, sort string, asc, random bool, commenter func(meta) (string, error), ceiling int) error {
func run(c client, org, query, sort string, asc, random bool, commenter func(meta) (string, error), ceiling int) error {
log.Printf("Searching: %s", query)
issues, err := c.FindIssues(query, sort, asc)
issues, err := c.FindIssuesWithOrg(org, query, sort, asc)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting, I don't remember this. The need for the org arg could be avoided by extracting it from the query, it should always have one of org:foo or repo:foo/bar

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the JWT tokes signed for the GH app are signed only for the installation ID which is per org, the query doesn't really matter afaik, so you can only query a single org in a single job with GH apps

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The point is right now you have to specify the same org in both the query and the CLI arg, it would be a better UX if we extracted the org from the query so it doesn't have to be duplicated into the CLI arg.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It has crossed my mind but:
a) you have to parse the query
b) the org can be implied be any of the repo: specs with the full name using /
c) with a) and b) together it would become complex and not intuitive (the error message can't just say use a single org: because of b) )

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think its very complex, you just have to extract all org: and repo: blocks and for the repo: blocks then extract the org. If overall you get exactly one org, use that. If not, error out mentioning that you got too many or none. This could also be directly in the github client method to get rid of the mandatory org arg.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this should be coupled to the GitHub search query syntax that can change over time. We'd always have to keep up. There is also set logic that you have to account for, like is:issue team:kubernetes/sig-testing -org:kubernetes-sigs

if err != nil {
return fmt.Errorf("search failed: %w", err)
}
Expand Down
4 changes: 2 additions & 2 deletions robots/commenter/main_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -222,7 +222,7 @@ func (c *fakeClient) CreateComment(owner, repo string, number int, comment strin
}

// Fakes searching for issues, using the same signature as github.Client
func (c *fakeClient) FindIssues(query, sort string, asc bool) ([]github.Issue, error) {
func (c *fakeClient) FindIssuesWithOrg(org, query, sort string, asc bool) ([]github.Issue, error) {
if strings.Contains(query, "error") {
return nil, errors.New(query)
}
Expand Down Expand Up @@ -314,7 +314,7 @@ func TestRun(t *testing.T) {
for _, tc := range cases {
ignoreSorting := ""
ignoreOrder := false
err := run(&tc.client, tc.query, ignoreSorting, ignoreOrder, false, makeCommenter(tc.comment, tc.template), tc.ceiling)
err := run(&tc.client, "", tc.query, ignoreSorting, ignoreOrder, false, makeCommenter(tc.comment, tc.template), tc.ceiling)
if tc.err && err == nil {
t.Errorf("%s: failed to received an error", tc.name)
continue
Expand Down