Better SCM CI Integration Workflows

Workflows
- Placeholder Variables
- Configuration File Location
Workflow Steps
Filters in Workflows
- Event Filter
- Branches Filter
Workflow Runs

Workflows

The .obs/workflows.yml file can contain multiple workflows with unique names

test_build:
  steps:
    - branch_package:
        source_project: OBS:Server:Unstable
        source_package: obs-server
        target_project: OBS:Server:Unstable:CI
  filters:
    event: pull_request
rebuild_master:
  steps:
    - rebuild_package:
        project: home:Admin
        package: ctris
  filters:
    event: push
    branches:
      only:
        - master

Placeholder Variables

With placeholder variables, workflows are now dynamic. Whenever a webhook event comes in, OBS downloads the workflows file and parses it. This is when the placeholder variables are replaced by the data they refer to in the webhook event payload.

Here's a list of supported placeholder variables and their mapping:

%{SCM_ORGANIZATION_NAME}: The name of the SCM organization, like openSUSE for the GitHub repository openSUSE/open-build-service
%{SCM_REPOSITORY_NAME}: The name of the SCM repository, like open-build-service for the GitHub repository openSUSE/open-build-service
%{SCM_PR_NUMBER}: The number of the pull request
%{SCM_COMMIT_SHA}: The SHA of the commit which was pushed

Example of workflows with some placeholder variables:

# test_build will branch a package based on the SCM repository name from which the webhook event came from.
test_build:
  steps:
    - branch_package:
        source_project: OBS:Server:Unstable
        source_package: %{SCM_REPOSITORY_NAME}
        target_project: OBS:Server:Unstable:CI
  filters:
    event: pull_request

Configuration File Location

Whenever a webhook event comes in, OBS expects the workflows configuration file to be at a specific location in the SCM repository which generated that webhook event. Anything else will prevent OBS from proceeding further. Users can customize in their workflow tokens where OBS expects this file to be located.

Below are the two options:

Path

Instead of looking for .obs/workflows.yml in the SCM repository, OBS will use the path defined in the workflow_configuration_path attribute of the workflow token. It can be anything, the user decides.
URL

OBS will download the workflows configuration file from the URL defined in the workflow_configuration_url attribute of the workflow token. The URL has to be reachable by OBS. The URL has precedence over the path if both are configured in the workflow token.

Workflow Steps

Branch a Package

Equivalent of osc branch

Providing the source project OBS:Server:Unstable, the source package obs-server and the target project OBS:Server:Unstable:CI will branch¹ the package OBS:Server:Unstable/obs-server:

to OBS:Server:Unstable:CI:$MY_SCM_ORG:$MY_SCM_PROJECT:PR-$MY_PR_NUMBER/obs-server for a pull request webhook event.
to OBS:Server:Unstable:CI/obs-server-$MY_COMMIT_SHA_OR_TAG_NAME for a push webhook event.

¹ Branching means creating a link, copying over files from the source package and creating configuring all repositories from the source project

workflow:
  steps:
    - branch_package:
        source_project: OBS:Server:Unstable
        source_package: obs-server
        target_project: OBS:Server:Unstable:CI

Link a Package to a Project

Equivalent of osc linkpac

Providing the source project OBS:Server:Unstable, the source package obs-server and the target project OBS:Server:Unstable:CI will link¹ the package OBS:Server:Unstable/obs-server:

to OBS:Server:Unstable:CI:$MY_SCM_ORG:$MY_SCM_PROJECT:PR-$MY_PR_NUMBER/obs-server for a pull request webhook event.
to OBS:Server:Unstable:CI/obs-server-$MY_COMMIT_SHA_OR_TAG_NAME for a push webhook event.

¹ Linking means creating a link pointing to the source package

workflow:
  steps:
    - link_package:
        source_project: OBS:Server:Unstable
        source_package: obs-server
        target_project: OBS:Server:Unstable:CI

Configure Repositories/Architectures for a Project

Providing the project OBS:Server:Unstable:CI and multiple repositories with each repository having a name, a target project, a target repository and their architectures will configure the project with the provided repositories and architectures.

Please note that the provided project has to be a project which was a target project from a previous step like link_package or branch_package. The target project will be:

OBS:Server:Unstable:CI:$MY_SCM_ORG:$MY_SCM_PROJECT:PR-$MY_PR_NUMBER for a pull request webhook event.
OBS:Server:Unstable:CI for a push webhook event.

workflow:
  steps:
    - configure_repositories:
        project: OBS:Server:Unstable:CI
        repositories:
          - name: openSUSE_Tumbleweed
            paths:
              - target_project: openSUSE:Factory
                target_repository: snapshot
              - target_project: openSUSE:Tumbleweed
                target_repository: standard
            architectures:
              - x86_64
              - i586
          - name: openSUSE_Leap_15.2
            paths:
              - target_project: openSUSE:Leap:15.2
                target_repository: standard
            architectures:
              - x86_64

Rebuild a Package

Equivalent of osc rebuild

Providing the project home:Admin and the package ctris will rebuild the package home:Admin/ctris.

This is possible only if the user of the token running the workflow has permissions to rebuild the package.

workflow:
  steps:
    - rebuild_package:
        project: home:Admin
        package: ctris

Set Flags

Providing the type build, the status enable and the project home:Admin will enable all builds:

for the home:Admin:$MY_SCM_ORG:$MY_SCM_PROJECT:PR-$MY_PR_NUMBER project when the webhook event is a pull request.
for the home:Admin when the webhook event is a push.

Please note that the provided project has to be a project which was a target project from a previous step like link_package or branch_package. Same for the package, it has to be branched/linked in a previous step.

The type has to be one of keys from FlagHelper::TYPES.

The status is either disable or enable.

Providing multiple flags is supported as noted in the YAML below.

workflow:
  steps:
    - set_flags:
        flags:
          - type: build
            status: enable
            project: home:Admin
          - type: publish
            status: disable
            project: home:Admin

The package, repository and architecture keys are all optional. When provided, they limit the flag to a certain package, repository or architecture.

So with the YAML provided below and a pull request event, builds of the home:Admin:$MY_SCM_ORG:$MY_SCM_PROJECT:PR-$MY_PR_NUMBER/ctris package will be disabled for the openSUSE_Tumbleweed repository and x86_64 architecture. For a push event, it's exactly the same, except for the package which is home:Admin/ctris-$MY_COMMIT_SHA_OR_TAG_NAME.

workflow:
  steps:
    - set_flags:
        flags:
          - type: build
            status: disable
            project: home:Admin
            package: ctris
            repository: openSUSE_Tumbleweed
            architecture: x86_64

Trigger Services

Equivalent of osc service

Providing the project home:Admin and the package ctris will trigger services of the package home:Admin/ctris.

This is possible only if the user of the token running the workflow has permissions to trigger services of the package.

workflow:
  steps:
    - trigger_services:
        project: home:Admin
        package: ctris

Submit Request

Equivalent of osc createrequest

Providing the source project home:Admin, the source package ctris and the target project openSUSE:Factory it will create a submit request from home:Admin/ctris to openSUSE:Factory/ctris.

workflow:
  steps:
    - submit_request:
        source_project: home:Admin
        source_package: ctris
        target_project: openSUSE:Factory
        target_package: ctris-2 (optional, will use source_package by default)
        description: 'This is the description of the submit request' (optional, will use a message containing SCM event details by default)

create the BsRequest if there is a pull_request event with the action open or a push/push_tag event
wait for services on source_package to finish, create a new BsReqest superseding all previous BsRequest from source_package to target_package if there is a pull_request event with action synchronized
revoke all BsRequest from source_package to target_package if there is a pull_request event with the action closed

Filters in Workflows

You can make use of the Branch and Event filters which restrict workflows to run only for or ignore certain branches/events.

Filters are defined in .obs/workflows.yml. Please refer to the subsections for details on each filter.

Here are some examples with the filters:

workflow:
  steps:
    - branch_package:
        source_project: home:jane_doe
        source_package: ctris
        target_project: games
  filters:
    event: pull_request
    branches:
      only:
        - master
        - staging

workflow:
  steps:
    - branch_package:
        source_project: home:jane_doe
        source_package: ctris
        target_project: games
  filters:
    event: push
    branches:
      ignore:
        - staging

Event Filter

Run workflow only for a specific GitHub/GitLab/Gitea event.

For GitHub events, see https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads.

For GitLab events, see https://docs.gitlab.com/ee/user/project/integrations/webhook_events.html.

For Gitea events. see https://docs.gitea.io/en-us/webhooks/

We use SCM-independent values for events:

pull_request for pull/merge requests events
push for push events related to commits
tag_push for push events related to tags

Example to run workflow only for a pull/merge request event from GitHub/GitLab/Gitea:

workflow:
  steps:
    - branch_package:
        source_project: home:jane_doe
        source_package: ctris
        target_project: games
  filters:
    event: pull_request

Branches Filter

⚠️ We do not support basic regular expression in the first version of branches filter. ⚠️

Matching target branches based on their names and run workflow only for those branches. Basic regular expression is supported with ^ (starting with) and $ (ending with).

Example to run workflow only for the target branch master, all target branches starting with staging and all target branches ending with final:

workflow:
  steps:
    - branch_package:
        source_project: home:jane_doe
        source_package: ctris
        target_project: games
  filters:
    branches:
      only:
        - master
        - ^staging
        - final$

Example to run workflow for all target branches, except master, those starting with staging and those ending with final:

workflow:
  steps:
    - branch_package:
        source_project: home:jane_doe
        source_package: ctris
        target_project: games
  filters:
    branches:
      ignore:
        - master
        - ^staging
        - final$

only has precedence over ignore, so if both are defined, ignore is not considered.

Workflow Runs

Every time we run a Workflow we create a WorkflowRun in the DB to capture what has happened. It captures:

various attributes extracted from the header/body like the name of the SCM (github), the name of the repository (openSUSE/open-build-service) and the name of the SCM event that triggered this workflow for debugging purposes.
our response to the webhook HTTP request and where we sent it to
the content of the workflow YAML file and where we got it from
all headers of the webhook HTTP request
the whole body of the webhook HTTP request
many associated WorkflowArtifactsPerStep object that captures what happened on OBS
many SCMStatusReport objects that capture what we reported back to the SCM

We expose some of this data in the UI to help people to debug their workflows themselves.

The rest of the attributes make it easy to debug what was going on. For instance what happened to the PR that introduced this feature?

WorkflowRun.where(scm_vendor: :github, repository_owner: 'openSUSE', repository_name: 'open-build-service', event_source_name: '14527').map { |workflow_run| [ workflow_run.created_at, workflow_run.hook_action ] }
=> => [[Wed, 21 Jun 2023 08:06:17.363658000 UTC +00:00, "opened"],
 [Wed, 21 Jun 2023 13:15:07.511126000 UTC +00:00, "synchronize"],
 [Wed, 21 Jun 2023 15:00:35.663343000 UTC +00:00, "synchronize"],
 ...
 [Mon, 26 Jun 2023 10:58:02.635911000 UTC +00:00, "synchronize"],
 [Mon, 26 Jun 2023 11:27:58.969067000 UTC +00:00, "closed"]]

Or who are our top 10 customers on github

WorkflowRun.where(scm_vendor: :github).group(:repository_owner).count.sort_by {|k,v| v}.reverse.first(10)

Better SCM CI Integration Workflows

Workflows

Placeholder Variables

Configuration File Location

Workflow Steps

Branch a Package

Link a Package to a Project

Configure Repositories/Architectures for a Project

Rebuild a Package

Set Flags

Trigger Services

Submit Request

Filters in Workflows

Event Filter

Branches Filter

Workflow Runs

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!