Radically simplify worker input/output ORT Result handling

[wkl3nk]

Context

What is the original scope of ORT Server (from https://github.com/eclipse-apoapsis/ort-server):

A scalable server implementation of the OSS Review Toolkit.

The Eclipse Apoapsis project's ORT Server is a standalone application to deploy the OSS Review Toolkit as a service in the cloud.

What does it mean:

• It allows to run the stages of the ORT pipeline in the cloud (Kubernetes) in a resource-efficient way and as a service.
• It offers a REST API to control it (trigger scan jobs, fetch reports)
• It offers a UI to control it

So the scope of ORT server is to provide an environment where ORT pipeline stages can be executed in a Kubernetes cloud, and have results, logs and reports stored in the cloud. Nothing less, nothing more.

Problem

So, isn't it as simple as taking the ORT results of the previous stage as input, process them in the current pipeline stage, and write the ORT result as output? The next stage then again takes this ORT result as input and so on ...

ORT Result Input --> stage: process in worker --> ORT Result output --> next stage ...

The problem is, in the meantime there is lots of code that:

• Reads the ORT results of the previous step out of a SQL database representation, performing additional tasks on this data, and feeding it then into the stage's worker.
• After the worker has finished, takes its ORT result output, and does post processing on it, deduplication, other kinds of things that are very complicated and then again stores the result in an individual way into the database.

This means, ORT server explicitly modifies results of pipeline stages in this or that way, instead to just transparently handle them unmodified as they are returned from ORT core.

This way, no longer you can be sure that the scan results from a traditional ORT pipeline are the same as the ones when you scan a repository with ORT Server.

Effects of this can already be seen in the UI: Issues and Vulnerabilities are displayed that differ from the ones that are generated by the ORT Reporters, because they operate on different data.

Proposal

• Radically get rid of any code that changes the ORT Result returned from ORT (core) in the workers. Just store the ORT Result unmodified.
• So the only thing that is required is code that a.) Stores an ORT Result in the SQL Database and b.) Reads an ORT Result from the SQL Database.

Benefit

• No more complex SQL handling code for post-processing ORT Results
• Chance for a simplified data model for ORT Result
• As ORT Results are no longer modified, the information displayed in the UI no longer deviates from the reports generated by the ORT reporters. ORT Server gets closer to ORT again.

Challenges

• Big changes in the database model are required
• UI: The BE code to provide entities to display may become harder, because you only have a bunch of ORT Result database representations in the database.