feat: langmgr and python #1091

sozercan · 2025-06-04T22:25:42Z

Signed-off-by: Sertac Ozercan [email protected]
Part of #147

Hold merge after current release

Describe the changes in this pull request using active verbs such as Add, Remove, Replace ...

Adds experimental app-level patching support with Python language manager

This PR introduces scaffolding for experimental support for patching application-level dependencies (libraries/packages) in addition to OS packages, with initial Python support.

Key Features

🔬 Experimental Feature

Requires COPA_EXPERIMENTAL=1 environment variable
Subject to breaking changes in future releases

📦 Package Type Filtering

New --pkg-types flag with values: os, library, os,library
Defaults to os (maintains backward compatibility)
Enables separate patching policies for OS vs application dependencies

🎯 Patch Level Control

New --library-patch-level flag with values: patch, minor, major
Defaults to patch for conservative updates
Implements preference ordering: patch > minor > major for safest upgrades

🐍 Python Support

Support for pip-managed Python packages. Other package managers are not considered for this initial PR.
Special handling for certifi (always latest version)

Example Usage

export COPA_EXPERIMENTAL=1
export IMAGE=mcr.microsoft.com/azure-cli:2.50.0 

# Scan for package vulnerabilities
trivy image --vuln-type os,library --ignore-unfixed -f json -o scan.json $IMAGE

# Patch only libraries with patch level updates only
copa patch -i $IMAGE -r scan.json --pkg-types library --library-patch-level patch

# Mixed OS and library patching with major level updates allowed
copa patch -i $IMAGE -r scan.json --pkg-types os,library --library-patch-level major

Signed-off-by: Sertac Ozercan <[email protected]>

codecov · 2025-06-05T00:38:36Z

Codecov Report

❌ Patch coverage is 57.05882% with 365 lines in your changes missing coverage. Please review.
✅ Project coverage is 45.40%. Comparing base (fb6b525) to head (c68a564).

Files with missing lines	Patch %	Lines
pkg/langmgr/python.go	54.38%	116 Missing and 14 partials ⚠️
pkg/patch/core.go	0.00%	62 Missing ⚠️
pkg/report/trivy.go	77.43%	44 Missing and 14 partials ⚠️
pkg/patch/cmd.go	16.07%	45 Missing and 2 partials ⚠️
pkg/patch/single.go	40.00%	35 Missing and 1 partial ⚠️
pkg/langmgr/langmgr.go	86.58%	8 Missing and 3 partials ⚠️
pkg/patch/patch.go	25.00%	7 Missing and 2 partials ⚠️
pkg/utils/utils.go	0.00%	9 Missing ⚠️
pkg/buildkit/buildkit.go	0.00%	1 Missing ⚠️
pkg/pkgmgr/dpkg.go	0.00%	1 Missing ⚠️
... and 1 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1091      +/-   ##
==========================================
+ Coverage   42.86%   45.40%   +2.53%     
==========================================
  Files          35       37       +2     
  Lines        4122     4898     +776     
==========================================
+ Hits         1767     2224     +457     
- Misses       2224     2509     +285     
- Partials      131      165      +34

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: Sertac Ozercan <[email protected]>

robert-cronin

just a few comments, otherwise LGTM!

robert-cronin · 2025-06-10T06:38:05Z

pkg/langmgr/python.go

+		var installCommands []string
+		for _, pkgArg := range installPkgArgs {
+			installCommands = append(installCommands,
+				fmt.Sprintf(`pip install %s || echo "WARN: pip install failed for %s"`, pkgArg, pkgArg))


should we do any validation/escaping here for the package names to prevent interpolation issues?

good point, added validation functions to validate names and version

robert-cronin · 2025-06-10T06:41:03Z

pkg/langmgr/python.go

+	var uniqueFailedPkgsList []string
+	if len(failedPackages) > 0 {
+		for _, pkgName := range failedPackages {
+			if !uniqueFailedPkgsMap[pkgName] {


it looks like this part might be duplicated a few times, perhaps we can pull out into a function like deduplicateStringSlice(input []string) []string?

added utils.DeduplicateStringSlice

robert-cronin · 2025-06-10T07:13:22Z

pkg/langmgr/python.go

+		}
+		installCmd := fmt.Sprintf(`sh -c '%s'`, strings.Join(installCommands, "; "))
+		pipInstalled = pm.config.ImageState.Run(
+			llb.Shlex(installCmd),


should we add a timeout for the pip install command?

by default pip has 15 seconds timeout. added a const for 300 seconds (5m), I am not sure if we want to make this configurable yet since we don't want too many knobs. maybe we can wait and see if someone wants a configurable timeout

pkg/langmgr/python.go

ashnamehrotra · 2025-06-09T19:23:54Z

pkg/langmgr/langmgr.go

+
+// GetLanguageManagers returns a list of all available language managers.
+// As new language managers are implemented, they should be added to this list.
+func GetLanguageManagers(config *buildkit.Config, workingFolder string) []LangManager {


is this function only for testing?

this is used in patch.go for getting the package managers (kinda equivalent to the GetPackageManager for OS packages), but we only have python in this PR

ashnamehrotra · 2025-06-09T20:47:02Z

website/docs/app-level-patching.md

+
+#### `minor` Level
+
+- **Allows**: `2.6.0` → `2.6.1` (preferred) or `2.7.0`


does this mean even if minor is set it will patch to 2.6.1 if both 2.6.1 and 2.7.0 are available?

correct, since focus is on CVE fixes while maintaining compatibility as much as possible. It'll only bump to 2.7.0 if 2.6.1 doesn't fix that CVE

pkg/patch/patch.go

Signed-off-by: Sertac Ozercan <[email protected]>

Copilot

Pull Request Overview

This PR introduces experimental support for app-level patching with Python package management in addition to existing OS package patching. The feature enables patching application-level dependencies using configurable patch levels and package type filtering.

Key changes:

Adds experimental Python language manager for patching pip-managed packages
Introduces --pkg-types flag to filter between OS and library packages
Implements --library-patch-level flag for controlling upgrade aggressiveness (patch/minor/major)
Updates manifest structure to separate OS and language updates

Reviewed Changes

Copilot reviewed 32 out of 33 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
pkg/langmgr/python.go	Core Python package manager implementation with pip integration
pkg/patch/patch.go	Enhanced patching logic to support language managers and package filtering
pkg/patch/cmd.go	New CLI flags for experimental package types and patch level control
pkg/report/trivy.go	Extended Trivy parser with optimal version selection and library patch level support
pkg/types/unversioned/types.go	Updated manifest structure separating OS and language updates
website/docs/app-level-patching.md	Comprehensive documentation for the new feature

pkg/langmgr/python.go

pkg/report/trivy.go

pkg/patch/patch.go

pkg/langmgr/python.go

pkg/report/trivy.go

Signed-off-by: Sertac Ozercan <[email protected]>

Signed-off-by: robert-cronin <[email protected]>

sozercan added 14 commits June 3, 2025 23:03

feat: langmgr and python

05549a9

Signed-off-by: Sertac Ozercan <[email protected]>

rename

02305b8

Signed-off-by: Sertac Ozercan <[email protected]>

pin

1889aaf

Signed-off-by: Sertac Ozercan <[email protected]>

validate

3b59147

Signed-off-by: Sertac Ozercan <[email protected]>

pip freeze all

44fe8e3

Signed-off-by: Sertac Ozercan <[email protected]>

lint

b8320dc

Signed-off-by: Sertac Ozercan <[email protected]>

handle multiple langs

b825081

Signed-off-by: Sertac Ozercan <[email protected]>

pkg-types

f85a523

Signed-off-by: Sertac Ozercan <[email protected]>

experimental flag

5a26993

Signed-off-by: Sertac Ozercan <[email protected]>

patch version pref

bb9751d

Signed-off-by: Sertac Ozercan <[email protected]>

address fail if no os vulns

6a93691

Signed-off-by: Sertac Ozercan <[email protected]>

handle ignore errors with multple pkgs

5f9ae1d

Signed-off-by: Sertac Ozercan <[email protected]>

single layer

c75aafe

Signed-off-by: Sertac Ozercan <[email protected]>

pick highest

e8a2b9b

Signed-off-by: Sertac Ozercan <[email protected]>

github-project-automation bot moved this to 🆕 New in Copacetic Workboard Jun 4, 2025

github-project-automation bot added this to Copacetic Workboard Jun 4, 2025

Merge branch 'main' into lang-python2

a5951c4

sozercan added 12 commits June 5, 2025 04:18

test image

c8f4720

Signed-off-by: Sertac Ozercan <[email protected]>

lint

ccd7ee1

Signed-off-by: Sertac Ozercan <[email protected]>

library requires scanner report

709d6ff

Signed-off-by: Sertac Ozercan <[email protected]>

tests

0a63715

Signed-off-by: Sertac Ozercan <[email protected]>

lint

dcbd79a

Signed-off-by: Sertac Ozercan <[email protected]>

lint

a44a48b

Signed-off-by: Sertac Ozercan <[email protected]>

library-patch-level

4daea82

Signed-off-by: Sertac Ozercan <[email protected]>

certifi exception

c35c12c

Signed-off-by: Sertac Ozercan <[email protected]>

e2e

5cd4751

Signed-off-by: Sertac Ozercan <[email protected]>

updates

ad9fc5b

Signed-off-by: Sertac Ozercan <[email protected]>

docs

9ab7b9e

Signed-off-by: Sertac Ozercan <[email protected]>

linty lint

4e2bf31

Signed-off-by: Sertac Ozercan <[email protected]>

sozercan marked this pull request as ready for review June 6, 2025 23:05

sozercan requested review from jeremyrickard and ashnamehrotra as code owners June 6, 2025 23:05

linty lint

7d0679f

Signed-off-by: Sertac Ozercan <[email protected]>

robert-cronin reviewed Jun 10, 2025

View reviewed changes

ashnamehrotra reviewed Jun 10, 2025

View reviewed changes

sozercan added 2 commits June 15, 2025 05:19

update

a612932

Signed-off-by: Sertac Ozercan <[email protected]>

update

f8b1533

Signed-off-by: Sertac Ozercan <[email protected]>

sozercan linked an issue Jun 25, 2025 that may be closed by this pull request

[REQ] support for app-level patching #147

Open

leodewang moved this from 🆕 New to 🏗 In progress in Copacetic Workboard Jun 27, 2025

leodewang moved this from 🏗 In progress to 👀 In review in Copacetic Workboard Jun 27, 2025

robert-cronin mentioned this pull request Jul 2, 2025

[REQ] Add python library patching #1092

Open

1 task

leodewang moved this from 👀 In review to 🏗 In progress in Copacetic Workboard Jul 2, 2025

robert-cronin added this to the v0.12.0 milestone Jul 14, 2025

address review comments

06cea75

Signed-off-by: Sertac Ozercan <[email protected]>

Copilot AI review requested due to automatic review settings July 25, 2025 22:09

Copilot AI reviewed Jul 25, 2025

View reviewed changes

sozercan added 2 commits July 25, 2025 22:55

update

6158d53

Signed-off-by: Sertac Ozercan <[email protected]>

copilot review

d610cfb

Signed-off-by: Sertac Ozercan <[email protected]>

sozercan marked this pull request as draft July 31, 2025 21:28

sozercan and others added 10 commits July 31, 2025 22:14

Merge branch 'main' into lang-python2

55f1f91

cleanup

d3a7153

Signed-off-by: Sertac Ozercan <[email protected]>

update

306de46

Signed-off-by: Sertac Ozercan <[email protected]>

lint

553ad42

Signed-off-by: Sertac Ozercan <[email protected]>

fix param order

7af5703

Signed-off-by: Sertac Ozercan <[email protected]>

fix test and const

46158f2

Signed-off-by: Sertac Ozercan <[email protected]>

update

4117ed7

Signed-off-by: Sertac Ozercan <[email protected]>

fix lll

36ae192

Signed-off-by: Sertac Ozercan <[email protected]>

fix state

9043cdf

Signed-off-by: Sertac Ozercan <[email protected]>

Merge branch 'main' into lang-python2

c68a564

Signed-off-by: robert-cronin <[email protected]>


		#### `minor` Level

		- Allows: `2.6.0` → `2.6.1` (preferred) or `2.7.0`

feat: langmgr and python #1091

Are you sure you want to change the base?

feat: langmgr and python #1091

Conversation

sozercan commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Adds experimental app-level patching support with Python language manager

Key Features

Example Usage

Uh oh!

codecov bot commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

robert-cronin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sozercan commented Jun 4, 2025 •

edited

Loading

codecov bot commented Jun 5, 2025 •

edited

Loading