Have functionality to remove components from SBOM if source code is already referenced

[tngraf]

In some organizations, license compliance for open source components is based on scanning the source code of these components for license and copyrights. Each source code should get processed only once. Having mutiple components in an SBOM that reference the same source code is not helpful. From the license compliance perspectice all these components could get reduced to a single component that references this shource code.

[gernot-h]

I can also imagine scenarios where this might make sense, at the latest when we want to add package entries with the new SW360 package portlet. So I hope this will be an optional feature or separate command?

[mag3me]

Can we understand this feature as an implicit granularity check?

Using the current granularity check I, as a user of capycli, can provide a list explicitly naming components I want to be joined into one component. The effect of capycli interpreting my list is that it removes components that match entries in my list from the SBOM and replaces them with (usually) fewer components using information from the original entries and the provided list.

I understand this feature to effectively do the same, but instead of using an explicit list, capycli detects components to be joined by looking at source code references.

[tngraf]

Can we understand this feature as an implicit granularity check?

Using the current granularity check I, as a user of capycli, can provide a list explicitly naming components I want to be joined into one component. The effect of capycli interpreting my list is that it removes components that match entries in my list from the SBOM and replaces them with (usually) fewer components using information from the original entries and the provided list.

I understand this feature to effectively do the same, but instead of using an explicit list, capycli detects components to be joined by looking at source code references.

Yes, this is correct. The granularity check is best on CaPyCLI or user provided lists of component names.
The idea of this feature is not based on component names, but on source code information. It will only work well if the software eco-system and the respective SBOM generator support information on the source code.

[gernot-h]

But I don't completely get how it should work. Say we have three components "foo-bin", "libfoo" and "foo-docs" in the SBOM, all referring to the "foo" sources (or source package). Do I get it right that you want to remove two of the three entries from the SBOM? If yes, which of them? Random?

From my pov, it would probably make more sense to have a converter binarySBOM -> sourceSBOM, so the SBOM would afterwards have only one component "foo".

[mag3me]

That is how the granularity check works at the moment. Using the granularity list we tell capycli that foo-bin is actually the package foo. And we do the same for libfooand foo-docs. As a result the three detected components will be removed from the SBOM. In their place there will be only one component in the resulting SBOM, named foo. It is called foo because that is the information we put into the input file for the granularity check.

I understand this new feature to not look at component names, but instead at source code information and from that infer that components need to be combined similar to what the granularity check does. Of course this can only work if there is adequate source code information and we don't always have adequate source code information available. But quite often we do and we can assume as capycli gets better at guessing/inferring source code information, this use case becomes more frequent.

In such situations I would appreciate it if capycli was implicitly able to combine foo-bin, libfoo and foo-docs into foo, without me having to augment my granularity file. Such a feature would obviously need some heuristics to guess the actual component name and work along the lines of this:

1. find components with similar source code references,
2. identify them as "actually being the same component",
3. guess the "correct" name,
4. modify the SBOM as the current granularity check would.

To guess the name we have all the information that comes with the SBOM. At the bare minimum there will be source code URLs or even file names, but we likely can also look at component names, purls and maybe even existing "matching" components in SW360. For some frequent use-cases it may even be useful to look into the referenced source code reference (e.g. Github project slugs) or even the actual source code (e.g. build system information).

[mag3me]

have a converter binarySBOM -> sourceSBOM, so the SBOM would afterwards have only one component "foo".

Any such feature could be optional instead of changing capycli's default behaviour. From a clearing/licensing perspective, the sourceSBOM would be sufficient and ensure the number of components grows slower in SW360. Why would we want to keep the binarySBOM around?

[gernot-h]

In such situations I would appreciate it if capycli was implicitly able to combine foo-bin, libfoo and foo-docs into foo, without me having to augment my granularity file. Such a feature would obviously need some heuristics to guess the actual component name and work along the lines of this:

1. find components with similar source code references,

2. identify them as "actually being the same component",

3. guess the "correct" name,

4. modify the SBOM as the current granularity check would.

To guess the name we have all the information that comes with the SBOM. At the bare minimum there will be source code URLs or even file names, but we likely can also look at component names, purls and maybe even existing "matching" components in SW360. For some frequent use-cases it may even be useful to look into the referenced source code reference (e.g. Github project slugs) or even the actual source code (e.g. build system information).

I exactly thought about such a feature, automatically concluding from binary packages to source packages without the user having to provide granularity files. But I don't think guessing is a good idea, so doing this "right" will probably only work with specific ecosystem knowledge. One of the problems is that with CycloneDX, there's no clear way to express information about sources, see also CycloneDX/specification#612. So to summarize, I'm quite unsure if there's a good way to implement this feature, neither the initial suggestion here nor my idea.

[gernot-h]

have a converter binarySBOM -> sourceSBOM, so the SBOM would afterwards have only one component "foo".

Any such feature could be optional instead of changing capycli's default behaviour. From a clearing/licensing perspective, the sourceSBOM would be sufficient and ensure the number of components grows slower in SW360. Why would we want to keep the binarySBOM around?

For us, experience has shown that a clearing workflow only based on source package information causes quite some confusion and friction as SBOMs from common tools (providing only binary package information) can't be mapped against existing clearing results.

That's one reason why the SW360 package portlet was added recently and there's the hope that modeling binary and source package information in the clearing workflow will help to reduce duplicates and wrong component modeling.

[mag3me]

But I don't think guessing is a good idea, so doing this "right" will probably only work with specific ecosystem knowledge.

Well not guessing as in randomly making stuff up. Guessing as in applying different heuristics based on information that may or may not be available. Yet, for a first implementation I would start out with having exactly one canonical implementation to derive a sensible component name. As a fallback, if capycli cannot determine a sensible name, it can always log about it and not perform the join/merge.

Longest substring/prefix searches on all candidates should repeatedly yield viable results. Results can definitely can be improved by using ecosystem knowledge and we should allow contributions for ecosystem specific heuristics. I think in many cases asking SW360 about matching components can yield good results as well. And we can always ask an AI ...

Ad sources:

One of the problems is that with CycloneDX, there's no clear way to express information about sources, see also

I suppose a good starting point for us would be source code references as stored in the sbom after matching components to SW360

[gernot-h]

Not sure if we want to go down the path of heuristics when filling an SBOM, that depends on the importance of correct component information for the rest of the workflow. There are often name clashes in different ecosystems, differences in package vs. project names etc etc. So I see a metionable risk we end up with the wrong component. If the component name is just decoration and all we care about in the end is a correct PURL and correct URLs (as we should do), then I'm fine with such heuristics.