Architecture Refactoring - How to integrate external Continuous Integration Services

[Mtze]

We're currently discussing how to simplify the integration of external CI systems. The existing implementation of ContinuousIntegrationService was originally designed to work with Bamboo - and still has some methods that are not needed anymore. With the new LocalCI system and potential other stateless external CI systems it would make sense to refactor this interface.

In this thread we wanna kick off the discussion how to do the transition.

Things that need to happen

(from @bensofficial and @Mtze perspective) - Feel free to critique and amend the suggestion!)

1. New version of ContinuousIntegrationService interface

   • Methods
     • build(participation) -> ID The main trigger should be one simple build method that handles all the necessary details internally
     • status(participation) -> Status: Returns the current state of the build for participation
       • Open questions:
         • Should we store the external build state in the db and have an endpoint for the external ci to update the state OR always relay the request to the external CI without storing the state.
         • To Handle the artemis client build updates we need to have an endpoint that allows a CI system to update the state of the build. e.g. POST /api/programming/public/exercise/<exerciseid>/submission/<submissionid>/status
     • health(): Returns the health of the external system

2. The implementations of JenkinsService and LocalCIService need to be adapted to adhere to the new interface

   • LocalCI should be rather easy to implement as the concept is already "stateless"
   • The jenkins integration could be a bit harder as jenkins requires a build plan. Lets discuss what options we have here

3. Update for the result handling - Suggestion: Split the ingest of build results (e.g. Tests) and the ingest of build logs.

   • Endpoint to add test results to a participation: POST /api/programming/public/exercise/<exerciseid>/submission/<submissionid>/result (new DTO Type: ExternalBuildResult)

     public record ExternalBuildResult(
          CompletionState completionState, // Indicator if Build did complete successfull, failed because of test cases, or failed with compilation errors,  
          ZonedDateTime buildStartDate, // Time the CI system started the build execution 
          ZonedDateTime buildCompletionDate, // Time the CI system completed the build 
          List<TestCaseResult> testCaseResults, // Flattened list of JUnit results 
          List<StaticCodeAnalysisReportDTO> staticCodeAnalysisReports, 
      ) implements BuildResultNotification, Serializable {}

     public enum CompletionState {
       BUILD_FAILED,
       TEST_FAILED,
       SUCCESSFUL
     }

   • Endpoint to add build log to a participation: POST /api/programming/public/exercise/<exerciseid>/submission/<submissionid>/log (existing DTO Type List<BuildLogDTO>)

[Mtze]

Feedback from SK and programming meeting:

• Solve the Trigger issue first - The Result handling should be a follow up
• Get the Jenkins People on board
• It might be necessary to have a delete method in the trigger service to get rid of the build plan afterwards (for e.g. Jenkins)
• One Jenkins build plan for each build - Otherwise the some concurrency could kick in

[dfuchss]

It will be highly important to be able to debug failed runs. For now, this was one of the big benefits of Jenkins. You can take a look into all created files after a CI run (esp. if the build failed).

[b-fein]

👍 for requirement to be able to debug Jenkins builds.

Making Jenkins ‘stateless’ and constantly creating and deleting build plans for each build would have multiple disadvantages in my opinion:

• debugging in Jenkins now impossible since no trace of the build plan is left after deletion
• additional system load on Jenkins for constant re-creation of build plans
  • not only CPU for many more API calls, but also state-keeping in Jenkins-internal database and the file system for build plan workspace creation/clean-up

To be able to debug the Jenkins builds, I guess the solution should not delete the build plans. However, I’m not sure how Jenkins will handle leaving thousands of build plans (since one is created per push per student) lying around.

---

As far as I remember, the Jenkins build trigger already checks if the build plan exists or not and creates it on-demand.

I would propose an approach that keeps the Jenkins state as it is right now, but removes the state-specific stuff from the general CI interfaces in Artemis and simplifies some other interactions:

• when a build is triggered: same as right now (create build plan if not exists or just trigger the build)

  • Jenkins ensures no race conditions between build plan creations/executions exist even when multiple pushes are directly after each other (Jenkins built-in queue)
  • the actual build script is fetched from the Artemis database, so compile & test execution specific state is saved in Jenkins -> no race conditions there, either

• remove all the scheduled build plan clean-up code (build plans are currently deleted for participants that already achieved 100% and/or did not push within the last few days)

  • not really needed
  • currently, there is a special participation state in Artemis when the build plan was already cleaned up -> no longer needed (not really needed right now either since build plan is automatically recreated)
  • only delete build plans when the exercise is deleted

• build plan clean-up

  • emit ‘exercise delete’ events somewhere in Artemis where the deletion is performed
  • we have a service implementation for Jenkins that listens to this event and removes the corresponding folder in Jenkins
  • goal: do not pollute generic CI-service interface with stuff the other stateless CI systems do not need

• permissions for editors/instructors to the build plans in Jenkins

  • note: student build plan access was planned initially, but never implemented
  • similar to clean-up: emit ‘course editors/instructors updated’ event
  • some Jenkins-related service in Artemis listens to this event and updates the exercise folders in Jenkins

• other stuff in Artemis that makes it more complicated than needed, but I forgot?

[krusche]

We definitely would like to remove the CI user management aspects in Artemis. This adds additional complexity and dependencies into the server code that increase the maintenance effort.

I understand that inspecting build results and files would be important to understand issues. I wonder if we could implement this without user management?

I don't think that creating and deleting build plans would lead to a high CPU usage. Creating those from scratch each time would allow specifying the build plan on demand and would avoid that Jenkins needs to fetch it from Artemis

[b-fein]

Result of discussion (KIT, Passau, TUM, Austrian Universities) today:

According to the knowledge in the current planning stage, the approach described below sounds feasible and sensible to all of us.

• Removing user management from Jenkins is okay.

  • In a first step, we as admins can give instructors local Jenkins accounts anyway.
  • Required second step, when the backchannel Jenkins->Artemis also gets reworked: Full Jenkins logs (i.e. including the Jenkins logs how it starts the containers) should be available in Artemis for instructors similar to as they are now already for LocalCI.

• Plan for future CI integration: Artemis provides a standard API it uses to talk with ‘CI-connector’ services running separately. We want to have a connector implementation for Jenkins.

  • Goals: easier maintenance of core Artemis
  • Artemis database still contains the actual build script (shell script in case of LocalCI, pipeline.groovy for Jenkins) so that it remains editable by the instructors, but the other templates around this (Jenkins build plan XML templates) would be part of the connector
    • (idea not discussed): maybe the connector can provide an API where it can tell Artemis ‘I support these programming languages’ and Artemis can then request the pipeline.groovy build script template from the connector? -> Jenkins-specific pipeline.groovy templates do not need to be part of Artemis main repository
  • Side-effect: connector implementations can evolve somewhat independently, separate maintainers for connectors can be assigned -> easier to get PRs merged

• Concrete goals for the Jenkins connector: copy/adapt the current Jenkins build plan management from Artemis but simplify a few cases

  • remove user management
  • current logic already assumes ‘if build plan does not exist then create; then trigger build’ -> can be kept without having to deal with database for connector keeping track of build plans in Jenkins
    • i.e. the connector would not create a build plan per push per student and delete it immediately afterwards -> connector deals with state rather than Artemis -> statelessness of CI system as far as Artemis is concerned
  • remove scheduled deletion tasks -> connector gets notified by Artemis once an exercise gets deleted -> only clean up exercise folder in Jenkins then

[Mtze]

Thanks for the summary! I was just about to also post the results of the discussion here :)

For now I'd say the administrator has to make sure that the "correct" type of CI system is present so we dont run into issues with the groovy files. In the long run i guess it would be good to also have an abstraction for these. We had a similar problem with bamboo and developed a system called aeolus which holds the build configuration and can create the application specific format from it (e.g. a Bamboo spec, or bash for local CI). In a future iteration we could think about only storing the build config in artemis - this would then also allow drop in replaces for CI systems - all of them would work the same (from an artemis pov) in the end.

[Mtze]

Here is a more detailed plan of the change.

Subsystem Decomposition

Here we compare the existing Internal API solution to the refactoring suggestion:

In the existing solution the Continuous Integration Service held the abstractions for BuildPlan management (many functions combined into one service here to keep it simple). There is a separate Continuous Integration Trigger Service which handles the build triggering. In the case of Jenkins there are many supporting Services that handle more detailed functionality. This is what we try to simplify. We suggest a change of the Continuous Integration Service to only expose three services: The existing health(), and 'getBuildStatus()' abstractions stay the same. We introduce a new build() service which is consumed by the Continuous Integration Trigger Service. This allows API compatability with the rest of Artemis but bundles are CI system specific details into only one service.

Scenarios

We identified a few scenarios that interact with stateful build settings:

To keep it simple - we omitted the Continuous Integration Service build redirection. It can be assumed that the Continuous Integration Trigger Service hides this redirect for these examples:

1. Programming Exercise Creation Scenario

This diagram shows the complete flow when a new programming exercise is created.

Existing Workflow

sequenceDiagram
    participant User as Instructor
    participant PER as ProgrammingExerciseResource
    participant PECS as ProgrammingExerciseCreationUpdateService
    participant PEPS as ProgrammingExerciseBuildPlanService
    participant CI as ContinuousIntegrationService
    participant CITS as ContinuousIntegrationTriggerService
    participant VCS as VersionControlService

    User->>PER: POST /programming-exercises (create exercise)
    PER->>PECS: createProgrammingExercise()
    
    Note over PECS: Setup repositories and participations
    PECS->>VCS: Create template/solution/test repositories
    
    PECS->>PEPS: setupBuildPlansForNewExercise()
    PEPS->>CI: createProjectForExercise() [TODO: Move to BuildPlanService]
    PEPS->>CI: createBuildPlanForExercise(TEMPLATE) [TODO: Move to BuildPlanService]
    PEPS->>CI: createBuildPlanForExercise(SOLUTION) [TODO: Move to BuildPlanService]
    PEPS->>CI: givePlanPermissions() [TODO: Move to PermissionService]
    
    Note over CI: Creates template and solution build plans
    PEPS->>CITS: triggerBuild(templateParticipation)
    PEPS->>CITS: triggerBuild(solutionParticipation)
    CITS-->>PEPS: Builds triggered
    CI-->>PEPS: Build plans created
    PEPS-->>PECS: Build plans setup complete
    PECS-->>PER: Exercise created
    PER-->>User: Exercise creation response

Loading

Refactored Workflow

sequenceDiagram
    participant User as Instructor
    participant PER as ProgrammingExerciseResource
    participant PECS as ProgrammingExerciseCreationUpdateService
    participant CITS as ContinuousIntegrationTriggerService
    participant VCS as VersionControlService

    User->>PER: POST /programming-exercises (create exercise)
    PER->>PECS: createProgrammingExercise()
    
    Note over PECS: Setup repositories and participations
    PECS->>VCS: Create template/solution/test repositories
    PECS->>CITS: triggerBuild(templateParticipation)
    PECS->>CITS: triggerBuild(solutionParticipation)
    CITS-->>PECS: Builds triggered
    PECS-->>PER: Exercise created
    PER-->>User: Exercise creation response

Loading

2. Student Participation Creation Scenario

This diagram shows what happens when a student starts working on a programming exercise.

Existing Workflow

sequenceDiagram
    participant Student as Student
    participant PR as ParticipationResource
    participant PS as ParticipationService
    participant CI as ContinuousIntegrationService
    participant VCS as VersionControlService

    Student->>PR: POST /participations (start exercise)
    PR->>PS: startExercise()
    
    Note over PS: Create student repository
    PS->>VCS: Copy template repository to student repo
    PS->>PS: configureRepository()
    
    Note over PS: Setup build plan for student
    PS->>PS: copyBuildPlan()
    PS->>CI: copyBuildPlan() [TODO: Move to BuildPlanService]
    CI-->>PS: Student build plan ID
    
    PS->>PS: configureBuildPlan()
    PS->>CI: configureBuildPlan() [TODO: Move to BuildPlanService]
    Note over CI: Configure student build plan with repo URL
    CI-->>PS: Build plan configured
    
    PS-->>PR: Participation initialized
    PR-->>Student: Repository and build plan ready

Loading

Refactored Workflow

sequenceDiagram
    participant Student as Student
    participant PR as ParticipationResource
    participant PS as ParticipationService
    participant VCS as VersionControlService

    Student->>PR: POST /participations (start exercise)
    PR->>PS: startExercise()
    
    Note over PS: Create student repository
    PS->>VCS: Copy template repository to student repo
    PS->>PS: configureRepository()
        
    PS-->>PR: Participation initialized
    PR-->>Student: Exercise ready

Loading

3. Programming Exercise Import Scenario

This diagram shows the scenario when importing an existing programming exercise.

Existing Workflow

sequenceDiagram
    participant User as Instructor
    participant PER as ProgrammingExerciseResource
    participant PEIS as ProgrammingExerciseImportService
    participant CI as ContinuousIntegrationService
    participant CITS as ContinuousIntegrationTriggerService

    User->>PER: POST /import-exercise
    PER->>PEIS: importProgrammingExercise()
    
    Note over PEIS: Setup new exercise structure
    PEIS->>CI: createProjectForExercise() [TODO: Move to BuildPlanService]
    PEIS->>CI: copyBuildPlan(template exercise → new exercise) [TODO: Move to BuildPlanService]
    PEIS->>CI: copyBuildPlan(solution exercise → new exercise) [TODO: Move to BuildPlanService]
    PEIS->>CI: enablePlan(template) [TODO: Move to BuildPlanService]
    PEIS->>CI: enablePlan(solution) [TODO: Move to BuildPlanService]
    PEIS->>CI: givePlanPermissions() [TODO: Move to PermissionService]
    
    Note over PEIS: Trigger builds for imported exercise
    PEIS->>CITS: triggerBuild(templateParticipation)
    PEIS->>CITS: triggerBuild(solutionParticipation)
    CITS-->>PEIS: Builds triggered
    
    CI-->>PEIS: Import completed
    PEIS-->>PER: Exercise imported
    PER-->>User: Import successful

Loading

Refactored Workflow

sequenceDiagram
    participant User as Instructor
    participant PER as ProgrammingExerciseResource
    participant PEIS as ProgrammingExerciseImportService
    participant CITS as ContinuousIntegrationTriggerService

    User->>PER: POST /import-exercise
    PER->>PEIS: importProgrammingExercise()
    
    Note over PEIS: Trigger builds for imported exercise
    PEIS->>CITS: triggerBuild(templateParticipation)
    PEIS->>CITS: triggerBuild(solutionParticipation)
    CITS-->>PEIS: Builds triggered
    
    PEIS-->>PER: Exercise imported
    PER-->>User: Import successful

Loading

4. Build Plan Configuration Update Scenario

This diagram shows what happens when the build configuration of an exercise is updated.

Existing Workflow

sequenceDiagram
    participant User as Instructor
    participant PER as ProgrammingExerciseResource
    participant PEPS as ProgrammingExerciseBuildPlanService
    participant CI as ContinuousIntegrationService

    User->>PER: PUT /programming-exercises/{id} (update build config)
    PER->>PEPS: updateBuildPlanForExercise()
    
    alt Build plan configuration changed
        Note over PEPS: Need to recreate build plans
        PEPS->>CI: deleteProject() [TODO: Move to BuildPlanService]
        PEPS->>CI: createProjectForExercise() [TODO: Move to BuildPlanService]
        PEPS->>CI: recreateBuildPlansForExercise() [TODO: Move to BuildPlanService]
        
        Note over CI: Recreate template and solution plans
        CI->>CI: deleteBuildPlan(template) [TODO: Move to BuildPlanService]
        CI->>CI: deleteBuildPlan(solution) [TODO: Move to BuildPlanService]
        CI->>CI: createBuildPlanForExercise(template) [TODO: Move to BuildPlanService]
        CI->>CI: createBuildPlanForExercise(solution) [TODO: Move to BuildPlanService]
        
        Note over PEPS: Reset all student build plan IDs
        PEPS->>PEPS: resetAllStudentBuildPlanIdsForExercise()
    end
    
    CI-->>PEPS: Update completed
    PEPS-->>PER: Build plan updated
    PER-->>User: Update successful

Loading

Refactored Workflow

sequenceDiagram
    participant User as Instructor
    participant PER as ProgrammingExerciseResource

    User->>PER: PUT /programming-exercises/{id} (update build config)

    PER-->>User: Update successful

Loading

5. Student Participation Resume Scenario

This diagram shows when a student resumes work on an exercise (e.g., after build plan was cleaned up).

Existing Workflow

sequenceDiagram
    participant Student as Student
    participant PR as ParticipationResource
    participant PS as ParticipationService
    participant CI as ContinuousIntegrationService

    Student->>PR: POST /resume-participation
    PR->>PS: resumeProgrammingExercise()
    
    Note over PS: Repository already exists, recreate build plan
    PS->>PS: copyBuildPlan()
    PS->>CI: copyBuildPlan() [TODO: Move to BuildPlanService]
    CI-->>PS: Build plan copied
    
    PS->>PS: configureBuildPlan()
    PS->>CI: configureBuildPlan() [TODO: Move to BuildPlanService]
    CI-->>PS: Build plan configured
    
    PS-->>PR: Participation resumed
    PR-->>Student: Exercise ready to continue

Loading

Refactored Workflow

We could even get rid of this functionality - Since we dont have to hold state anymore, it would be an option to never get a Exercise into the "Resume" state

sequenceDiagram
    participant Student as Student
    participant PR as ParticipationResource
    participant PS as ParticipationService


    Student->>PR: POST /resume-participation
    PR->>PS: resumeProgrammingExercise()
    
    PS-->>PR: Participation resumed
    PR-->>Student: Exercise ready to continue

Loading

6. Exercise Deletion Scenario

This diagram shows the cleanup process when a programming exercise is deleted.

Existing Workflow

sequenceDiagram
    participant User as Instructor
    participant PER as ProgrammingExerciseResource
    participant PEDS as ProgrammingExerciseDeletionService
    participant CI as ContinuousIntegrationService
    participant VCS as VersionControlService

    User->>PER: DELETE /programming-exercises/{id}
    PER->>PEDS: deleteProgrammingExercise()
    
    Note over PEDS: Clean up CI resources
    PEDS->>CI: deleteBuildPlan(template) [TODO: Move to BuildPlanService]
    PEDS->>CI: deleteBuildPlan(solution) [TODO: Move to BuildPlanService]
    PEDS->>CI: deleteProject() [TODO: Move to BuildPlanService]
    
    Note over PEDS: Clean up VCS resources
    PEDS->>VCS: Delete repositories
    
    CI-->>PEDS: CI cleanup complete
    VCS-->>PEDS: VCS cleanup complete
    PEDS-->>PER: Exercise deleted
    PER-->>User: Deletion successful

Loading

Refactored Workflow

sequenceDiagram
    participant User as Instructor
    participant PER as ProgrammingExerciseResource
    participant PEDS as ProgrammingExerciseDeletionService
    participant VCS as VersionControlService

    User->>PER: DELETE /programming-exercises/{id}
    PER->>PEDS: deleteProgrammingExercise()
    
    Note over PEDS: Clean up VCS resources
    PEDS->>VCS: Delete repositories
    
    VCS-->>PEDS: VCS cleanup complete
    PEDS-->>PER: Exercise deleted
    PER-->>User: Deletion successful

Loading

7. Build Plan Repository Update Scenario

This diagram shows when repository URLs need to be updated in build plans.

Existing Workflow

sequenceDiagram
    participant System as System Process
    participant PS as ParticipationService
    participant CI as ContinuousIntegrationService

    System->>PS: updateParticipationRepository()
    PS->>CI: updatePlanRepository() [TODO: Move to BuildPlanService]
    
    Note over CI: Update repository URL in build plan
    CI->>CI: Update build plan configuration
    CI->>CI: enablePlan() [TODO: Move to BuildPlanService]
    
    CI-->>PS: Repository updated
    PS-->>System: Update complete

Loading

Refactored Workflow

We should discuss if this is still needed in the stateless usecase

sequenceDiagram
    participant System as System Process
    participant PS as ParticipationService
    System->>PS: updateParticipationRepository()
    PS-->>System: Update complete

Loading

8. Consistency Check Scenario

This diagram shows the consistency checking process for exercises.

Existing Workflow

sequenceDiagram
    participant System as Background Process
    participant CCS as ConsistencyCheckService
    participant CI as ContinuousIntegrationService
    participant VCS as VersionControlService

    System->>CCS: checkConsistency()
    
    Note over CCS: Check VCS consistency
    CCS->>VCS: checkIfProjectExists()
    VCS-->>CCS: Project status
    
    Note over CCS: Check CI consistency
    CCS->>CI: checkIfBuildPlanExists(template) [TODO: Move to BuildPlanService]
    CI-->>CCS: Template build plan status
    
    CCS->>CI: checkIfBuildPlanExists(solution) [TODO: Move to BuildPlanService]
    CI-->>CCS: Solution build plan status
    
    CCS-->>System: Consistency report

Loading

Refactored Workflow

sequenceDiagram
    participant System as Background Process
    participant CCS as ConsistencyCheckService
    participant VCS as VersionControlService

    System->>CCS: checkConsistency()
    
    Note over CCS: Check VCS consistency
    CCS->>VCS: checkIfProjectExists()
    VCS-->>CCS: Project status
    
    CCS-->>System: Consistency report

Loading

[Mtze]

I started implementing a POC here: https://github.com/ls1intum/Artemis/tree/feat/stateless-ci-interface