Better SCM CI Integration Overview

Monitoring
Workflows inside the App
Testing In Development Environment

NOTE: For now we only support local packages.

Monitoring

This feature is monitored on OBS and IBS.

Workflows inside the App

Workflow triggering

SCM triggers a webhook pointing to our trigger/workflow endpoint (someone opened or updated a PR).
Inside TriggerWorkflowController:
- We extract the event, SCM, and payload from the incoming request.
- Calls Token::Workflow model and passes the extracted data from the request.
Token::Workflow
- Extract data from the webhook payload using the TriggerControllerService::ScmExtractor.
- Download the .obs/workflows.yml file from the target branch, not from the Pull request, using the Workflows::YAMLDownloader.
- Call the Workflows::YAMLToWorkflowsService which returns a collection of Workflow objects for each workflow defined in the configuration file.
- From each Workflow object we get the corresponding steps (Workflow::Step::BranchPackageStep, Workflow::Step::LinkPackageStep and Workflow::Step::ConfigureRepositoriesStep for now) and execute them.
- An Event::Subscription is created for the token (storing the token ID in the event_subscriptions.token_id column) and in that Event::Subscription, we store the required information to be able to report back to the SCM (repository owner, etc... see below in Reporting Back to SCM).
- Report back to SCM the "pending" state using SCMStatusReporter service.

NOTE: To make the service run on a linked package (link_package step) we had to make it a "project service" by adding a package named _project into the target project. The package contains a _service file with a correct definition of a service.

Reporting back final build results

Whenever a build finishes in the Backend, a new Event::BuildFail or Event::BuildSuccess is created and a new ReportToScmJob is queued in scm queue. Follow this link for a better understanding of the Events system.
- ReportToScmJob inherits from CreateJob.
- CreateJob is only for jobs dealing with events, since its perform method expects an event_id.
- For the jobs to be created, event classes like Event::BuildFail and Event::BuildSuccess need to have the following code: create_jobs :report_to_scm_job. This will then be picked up in the after_create :perform_create_jobs callback in the Event::Base model.
- If the provided symbol in the create_jobs method is a valid job inheriting from CreateJob, then the corresponding job will be queued.
- The scm queue service is defined in systemd/obs-delayedjob-queue-scm.service
The ReportToScmJob calls SCMStatusReporter service and sets the status on the corresponding commit.
- We pass the event_id to the ReportToScmJob. Using the event_id we are fetching the event object, we search for the event subscriptions that match the given event object (eventtype and package), loop over them and call SCMStatusReporter for each one.

Workflows

The .obs/workflows.yml file can contain multiple workflows with unique names

test_build:
  steps:
    - branch_package
        source_project: OBS:Server:Unstable
        source_package: obs-server
        target_project: OBS:Server:Unstable:CI
  filters:
    event: pull_request
rebuild_master:
  steps:
    - rebuild_package
        project: home:Admin
        package: ctris
  filters:
    event: push
    branches:
      only:
        - master

Workflow Steps

Branch a Package

Providing the source project OBS:Server:Unstable, the source package obs-server and the target project OBS:Server:Unstable:CI will branch the package OBS:Server:Unstable/obs-server:

to OBS:Server:Unstable:CI:$MY_SCM_ORG:$MY_SCM_PROJECT:PR-$MY_PR_NUMBER/obs-server for a pull request webhook event.
to OBS:Server:Unstable:CI/obs-server-$MY_COMMIT_SHA_OR_TAG_NAME for a push webhook event.

Check if the token user is allowed to branch a package in the provided target project.

workflow:
  steps:
    - branch_package:
        source_project: OBS:Server:Unstable
        source_package: obs-server
        target_project: OBS:Server:Unstable:CI

Link a Package to a Project

Equivalent of osc linkpac

Providing the source project OBS:Server:Unstable, the source package obs-server and the target project OBS:Server:Unstable:CI will link the package OBS:Server:Unstable/obs-server:

to OBS:Server:Unstable:CI:$MY_SCM_ORG:$MY_SCM_PROJECT:PR-$MY_PR_NUMBER/obs-server for a pull request webhook event.
to OBS:Server:Unstable:CI/obs-server-$MY_COMMIT_SHA_OR_TAG_NAME for a push webhook event.

Check if the token user is allowed to link a package in this target project.

workflow:
  steps:
    - link_package:
        source_project: OBS:Server:Unstable
        source_package: obs-server
        target_project: OBS:Server:Unstable:CI

Configure Repositories/Architectures for a Project

Providing the project OBS:Server:Unstable:CI and multiple repositories with each repository having a name, a target project, a target repository and their architectures will configure the project with the provided repositories and architectures.

Please note that the provided project has to be a project which was a target project from a previous step like link_package or branch_package. The target project will be:

OBS:Server:Unstable:CI:$MY_SCM_ORG:$MY_SCM_PROJECT:PR-$MY_PR_NUMBER for a pull request webhook event.
OBS:Server:Unstable:CI for a push webhook event.

Check if the token user is allowed to configure repositories/architectures for this project.

workflow:
  steps:
    - configure_repositories:
        project: OBS:Server:Unstable:CI
        repositories:
          - name: openSUSE_Tumbleweed
            paths:
              - target_project: openSUSE:Factory
                target_repository: snapshot
              - target_project: openSUSE:Tumbleweed
                target_repository: standard
            architectures:
              - x86_64
              - i586
          - name: openSUSE_Leap_15.2
            paths:
              - target_project: openSUSE:Leap:15.2
                target_repository: standard
            architectures:
              - x86_64

Rebuild a Package

Providing the project home:Admin and the package ctris will rebuild the package home:Admin/ctris.

This is possible only if the user of the token running the workflow has permissions to rebuild the package.

workflow:
  steps:
    - rebuild_package:
        project: home:Admin
        package: ctris

Set Flags

Providing the type build, the status enable and the project home:Admin will enable all builds:

for the home:Admin:$MY_SCM_ORG:$MY_SCM_PROJECT:PR-$MY_PR_NUMBER project when the webhook event is a pull request.
for the home:Admin when the webhook event is a push.

Please note that the provided project has to be a project which was a target project from a previous step like link_package or branch_package. Same for the package, it has to be branched/linked in a previous step.

The type has to be one of keys from FlagHelper::TYPES.

The status is either disable or enable.

Providing multiple flags is supported as noted in the YAML below.

workflow:
  steps:
    - set_flags:
        flags:
          - type: build
            status: enable
            project: home:Admin
          - type: publish
            status: disable
            project: home:Admin

The package, repository and architecture keys are all optional. When provided, they limit the flag to a certain package, repository or architecture.

So with the YAML provided below and a pull request event, builds of the home:Admin:$MY_SCM_ORG:$MY_SCM_PROJECT:PR-$MY_PR_NUMBER/ctris package will be disabled for the openSUSE_Tumbleweed repository and x86_64 architecture. For a push event, it's exactly the same, except for the package which is home:Admin/ctris-$MY_COMMIT_SHA_OR_TAG_NAME.

workflow:
  steps:
    - set_flags:
        flags:
          - type: build
            status: disable
            project: home:Admin
            package: ctris
            repository: openSUSE_Tumbleweed
            architecture: x86_64

Trigger Services

Providing the project home:Admin and the package ctris will trigger services of the package home:Admin/ctris.

This is possible only if the user of the token running the workflow has permissions to trigger services of the package.

workflow:
  steps:
    - trigger_services:
        project: home:Admin
        package: ctris

Filters in Workflows

There are two types of filters:

Architectures and Repositories filters which affect what we report back to the SCM
Branch and Event filters which restrict workflows to run only for or ignore certain branches/events

Filters are defined in .obs/workflows.yml. Please refer to the subsections for details on each filter.

Here's an example with all filters:

workflow:
  steps:
    - branch_package:
        source_project: home:jane_doe
        source_package: ctris
        target_project: games
  filters:
    event: pull_request
    branches:
      only:
        - master
        - staging
    architectures:
      ignore:
        - s390x
        - ia64
    repositories:
      only:
        - openSUSE_Tumbleweed

Event Filter

Run workflow only for a specific GitHub/GitLab event.

For GitHub events, see https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads.

For GitLab events, see https://docs.gitlab.com/ee/user/project/integrations/webhook_events.html.

We use SCM-independent values for events:

pull_request for pull/merge requests events
push for push events related to commits
tag_push for push events related to tags

Example to run workflow only for a pull/merge request event from GitHub/GitLab:

workflow:
  steps:
    - branch_package:
        source_project: home:jane_doe
        source_package: ctris
        target_project: games
  filters:
    event: pull_request

Branches Filter

⚠️ We do not support basic regular expression in the first version of branches filter. ⚠️

Matching target branches based on their names and run workflow only for those branches. Basic regular expression is supported with ^ (starting with) and $ (ending with).

Example to run workflow only for the target branch master, all target branches starting with staging and all target branches ending with final:

workflow:
  steps:
    - branch_package:
        source_project: home:jane_doe
        source_package: ctris
        target_project: games
  filters:
    branches:
      only:
        - master
        - ^staging
        - final$

Example to run workflow for all target branches, except master, those starting with staging and those ending with final:

workflow:
  steps:
    - branch_package:
        source_project: home:jane_doe
        source_package: ctris
        target_project: games
  filters:
    branches:
      ignore:
        - master
        - ^staging
        - final$

only has precedence over ignore, so if both are defined, ignore is not considered.

Architectures Filter

Matching architectures based on their names and report back to the SCM only for those architectures.

Example of a workflow reporting back to the SCM only for architectures s390x and x86_64:

workflow:
  steps:
    - branch_package:
        source_project: home:jane_doe
        source_package: ctris
        target_project: games
  filters:
    architectures:
      only:
        - s390x
        - x86_64

Example of a workflow reporting back to the SCM for all architectures, except s390x and x86_64:

workflow:
  steps:
    - branch_package:
        source_project: home:jane_doe
        source_package: ctris
        target_project: games
  filters:
    architectures:
      ignore:
        - s390x
        - x86_64

only has precedence over ignore, so if both are defined, ignore is not considered.

Repositories Filter

Matching repositories based on their names and report back to the SCM only for those repositories.

Example of a workflow reporting back to the SCM only for repositories openSUSE_Tumbleweed and CentOS_8:

workflow:
  steps:
    - branch_package:
        source_project: home:jane_doe
        source_package: ctris
        target_project: games
  filters:
    repositories:
      only:
        - openSUSE_Tumbleweed
        - CentOS_8

Example of a workflow reporting back to the SCM for all repositories, except openSUSE_Tumbleweed:

workflow:
  steps:
    - branch_package:
        source_project: home:jane_doe
        source_package: ctris
        target_project: games
  filters:
    repositories:
      ignore:
        - openSUSE_Tumbleweed

only has precedence over ignore, so if both are defined, ignore is not considered.

Workflow Runs

Whenever a workflow token is triggered by an incoming webhook, a workflow run is created to track what is happening, thus helping users understand how their workflow behaves. In addition to the headers and payload of the incoming webhook, workflow runs store the response we send back to the SCM and to which URL it is sent. Finally, the status is saved, so users know if the workflow run succeeded or potentially failed due to an error.

Testing In Development Environment

Most of the functionality can be tested by faking the payload sent by an SCM event. You can simply run a curl command against a local endpoint.

Create a personal access token on GitHub with the public_repo scope.
Create a workflow token on OBS with the scm_token containing the token created on GitHub.
Take the payload of a recent delivery from one of the OBS webhooks and save it in a file like payload.json.
Create a workflows.yml file under src/api/ with the workflows and steps you want to test (you can use our OBS workflow if needed). Change the method Workflows::YAMLDownloader#call to instead return File.open('workflows.yml'). This is to use the workflows.yml you created.
In your local OBS instance, create the project/package/repositories/architectures needed for the workflows.yml you created in the previous step.
Change the method TriggerControllerService::TokenExtractor#valid_signature? to return true. This is to circumvent the authentication, unless you want to test that.

Send a POST request with curl. Change the ALL CAPS to the according values. See the example below:

GitHub:

curl -X POST 'http://localhost:3000/trigger/workflow?id=YOUR_WORKFLOW_TOKEN_ID' --data @payload.json --header 'X-GitHub-Event: SCM_EVENT_YOU_WANT' --header 'Content-Type: application/json' --header 'X-Hub-Signature-256: sha256=YOUR_WORKFLOW_TOKEN_STRING'

GitLab:

curl -X POST 'http://localhost:3000/trigger/workflow?id=YOUR_WORKFLOW_TOKEN_ID' --data @payload.json --header 'X-Gitlab-Event: SCM_EVENT_YOU_WANT' --header 'Content-Type: application/json' --header 'X-Gitlab-Token: YOUR_WORKFLOW_TOKEN_STRING'

With ngrok

In case it is extremely needed to connect your local application with a real SCM, use ngrok as we describe here:

Warning: always log out of your VPN before starting using ngrok.

Create an account in https://ngrok.com and follow the instructions, which are basically:
- Download ngrok.zip and unzip it
- Add the authtoken with ./ngrok authtoken <authtoken>
Run ngrok http 3000 to set the tunnel.
A UI will be displayed in your terminal with the public URL of your tunnel, copy it.
Add the host of the public URL to config/environments/development.rb. Example: config.hosts << "2406-79-153-113-247.ngrok.io"
Restart the local server.
Both localhost:3000 and the public URL should be accessible from your browser.
Follow this guide to integrate your locally-running application with GitHub, keeping this in mind:
- Create the OBS workflow token locally
- The "Payload URL" field in GitHub webhook should look like: https://2406-79-153-113-247.ngrok.io/trigger/workflow?id=0
Open a new PR towards the GitHub repository where you configured the webhook, so you will see that the new subproject has been created in the application running locally.

Better SCM CI Integration Overview

Monitoring

Workflows inside the App

Workflow triggering

Reporting back final build results

Workflows

Workflow Steps

Branch a Package

Link a Package to a Project

Configure Repositories/Architectures for a Project

Rebuild a Package

Set Flags

Trigger Services

Filters in Workflows

Event Filter

Branches Filter

Architectures Filter

Repositories Filter

Workflow Runs

Testing In Development Environment

With ngrok

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!