|
| 1 | +#### Fixing CVEs |
| 2 | + |
| 3 | +**Before starting, user must:** |
| 4 | + |
| 5 | +Install/update grype: |
| 6 | +```bash |
| 7 | +grype version || curl -sSfL https://get.anchore.io/grype | sudo sh -s -- -b /usr/local/bin |
| 8 | +``` |
| 9 | + |
| 10 | +Fetch latest changes: |
| 11 | +```bash |
| 12 | +git fetch |
| 13 | +``` |
| 14 | + |
| 15 | +##### 0. Update Vulnerability Database |
| 16 | + |
| 17 | +```bash |
| 18 | +grype db update |
| 19 | +``` |
| 20 | + |
| 21 | +##### 1. Update Branch and Create Fix Branch |
| 22 | + |
| 23 | +```bash |
| 24 | +# Switch to target branch and update to latest |
| 25 | +git checkout release-0.X |
| 26 | +git merge --ff-only origin/release-0.X |
| 27 | + |
| 28 | +# Create fix branch (add -v2, -v3 etc if branch exists; don't delete old versions) |
| 29 | +git checkout -b fix-0.X-cves-YYYY-MM-DD |
| 30 | +``` |
| 31 | + |
| 32 | +Replace `0.X`: `release-0.21` → `0.21`, `devel` → `devel`. |
| 33 | + |
| 34 | +One commit per package (may fix multiple CVEs), one PR total. |
| 35 | + |
| 36 | +##### 2. Scan |
| 37 | + |
| 38 | +```bash |
| 39 | +grype . --config .grype.yaml -o table |
| 40 | +``` |
| 41 | + |
| 42 | +Ignore warning "no explicit name and version provided". |
| 43 | + |
| 44 | +For each CVE, note: |
| 45 | +- Package (**NAME**) |
| 46 | +- Version (**FIXED-IN**) |
| 47 | +- CVE ID (**VULNERABILITY**) |
| 48 | + |
| 49 | +**Note**: Same package may appear multiple times with different versions (e.g., quic-go v0.48.2 in coredns, v0.54.0 in main). Treat each as separate fix. |
| 50 | + |
| 51 | +##### 3. Locate Package |
| 52 | + |
| 53 | +```bash |
| 54 | +grep -Fl [package] go.mod coredns/go.mod tools/go.mod 2>/dev/null |
| 55 | +``` |
| 56 | + |
| 57 | +**Check for replace directives**: |
| 58 | +```bash |
| 59 | +grep "replace.*$(basename [package])" go.mod coredns/go.mod tools/go.mod 2>/dev/null |
| 60 | +``` |
| 61 | + |
| 62 | +If found, check git history why it was added. If obsolete, remove now. Otherwise update to safe version in Step 4: |
| 63 | +```bash |
| 64 | +# Check history |
| 65 | +git log -p --all -S "replace.*$(basename [package])" -- [module]/go.mod | head -50 |
| 66 | + |
| 67 | +# Remove if obsolete |
| 68 | +sed -i '/replace.*[package]/d; /Fixes CVE-/d' [module]/go.mod |
| 69 | +``` |
| 70 | + |
| 71 | +**Check for parent-child dependencies:** |
| 72 | + |
| 73 | +If multiple packages in the same module have CVEs, check if one depends on another: `cd [module] && go mod graph | grep [package]`. If parent (e.g., CoreDNS depends on quic-go) also has CVE, fix parent first. Parent upgrade often brings newer child, eliminating separate fix. |
| 74 | + |
| 75 | +##### 4. Update |
| 76 | + |
| 77 | +**If Step 3 found package in single module:** |
| 78 | + |
| 79 | +```bash |
| 80 | +# coredns/go.mod only |
| 81 | +cd coredns && go get [package]@v[version] && go mod tidy && cd .. |
| 82 | + |
| 83 | +# tools/go.mod only |
| 84 | +cd tools && go get [package]@v[version] && go mod tidy && cd .. |
| 85 | + |
| 86 | +# go.mod only |
| 87 | +go get [package]@v[version] && go mod tidy |
| 88 | +``` |
| 89 | + |
| 90 | +If multiple CVEs have different FIXED-IN versions, use the highest. |
| 91 | + |
| 92 | +**If Step 3 found package in multiple modules:** |
| 93 | + |
| 94 | +Multi-module packages require verification between updates to avoid downgrades. |
| 95 | + |
| 96 | +1. If package in main go.mod (and maybe submodules too): |
| 97 | + ```bash |
| 98 | + go get [package]@v[version] && go mod tidy |
| 99 | + ``` |
| 100 | + Then proceed to Step 5, Step 6. If Step 6 shows CVE still present, update one submodule at a time, repeating Steps 5-6 after each update. |
| 101 | + |
| 102 | +2. If package only in submodules (e.g., both coredns and tools): |
| 103 | + ```bash |
| 104 | + cd coredns && go get [package]@v[version] && go mod tidy && cd .. |
| 105 | + ``` |
| 106 | + Then proceed to Step 5, Step 6. If Step 6 shows CVE still present, update next submodule (tools), repeating Steps 5-6. |
| 107 | + |
| 108 | +Submodules may have different (non-vulnerable) versions. Check `git diff` for unexpected downgrades (e.g., CoreDNS version changes). |
| 109 | + |
| 110 | +**On stable release branches, if `go get` upgrades:** |
| 111 | +- Go minor version (1.X => 1.Y) |
| 112 | +- K8s minor version (v0.A => v0.B) |
| 113 | + |
| 114 | +Revert changes. Low/Medium CVEs: skip to Step 9. High/Critical: consult team. |
| 115 | + |
| 116 | +Remove `toolchain` lines and extra blank lines added by `go mod tidy`: |
| 117 | + |
| 118 | +```bash |
| 119 | +sed -i '/^toolchain/d' go.mod coredns/go.mod tools/go.mod |
| 120 | +sed -i '/^$/{N;/^\n$/s/\n//;}' go.mod coredns/go.mod tools/go.mod |
| 121 | +``` |
| 122 | + |
| 123 | +Verify changes: |
| 124 | + |
| 125 | +```bash |
| 126 | +git diff |
| 127 | +``` |
| 128 | + |
| 129 | +Expected: Dependency file updates only. |
| 130 | + |
| 131 | +##### 5. Clean Binaries |
| 132 | + |
| 133 | +```bash |
| 134 | +make clean-generated |
| 135 | +``` |
| 136 | + |
| 137 | +Removes build artifacts to avoid false positives. |
| 138 | + |
| 139 | +Network errors: see Common Issues. |
| 140 | + |
| 141 | +##### 6. Verify |
| 142 | + |
| 143 | +```bash |
| 144 | +grype . --config .grype.yaml -o table |
| 145 | +``` |
| 146 | + |
| 147 | +CVE(s) for this package should no longer appear. |
| 148 | + |
| 149 | +**If CVE persists**: Double-check you used the correct version from Step 2 FIXED-IN column. If version is correct but CVE persists, the replace directive should have been caught in Step 3 - recheck Step 3. |
| 150 | + |
| 151 | +##### 7. Verify Build |
| 152 | + |
| 153 | +```bash |
| 154 | +make unit |
| 155 | +``` |
| 156 | + |
| 157 | +**Note**: When fixing multiple packages, skip this step for each package and run once at end. |
| 158 | + |
| 159 | +Build errors may indicate incompatible dependency versions for this branch. |
| 160 | + |
| 161 | +##### 8. Commit |
| 162 | + |
| 163 | +```bash |
| 164 | +# Stage dependency files |
| 165 | +git add go.mod go.sum coredns/go.mod coredns/go.sum tools/go.mod tools/go.sum |
| 166 | + |
| 167 | +# Verify what will be committed |
| 168 | +git diff --staged --stat |
| 169 | +``` |
| 170 | + |
| 171 | +Expected: Only go.mod/go.sum files (for modules that changed). |
| 172 | + |
| 173 | +Follow commit templates in @.agents/commit-templates.md (CVE Fixes section). Use "in /[module]" format if only coredns or tools files changed. |
| 174 | + |
| 175 | +After each commit, rescan to check for newly introduced CVEs: |
| 176 | +```bash |
| 177 | +grype . --config .grype.yaml -o table |
| 178 | +``` |
| 179 | + |
| 180 | +Repeat steps 3-8 for each remaining package with CVEs. |
| 181 | + |
| 182 | +##### 9. When to Ignore CVEs |
| 183 | + |
| 184 | +If fix requires breaking changes (Go version, K8s major version, incompatible APIs), consider: |
| 185 | +- CVE severity (Low/Medium vs High/Critical) |
| 186 | +- Cost/risk of breaking stable branch dependencies |
| 187 | + |
| 188 | +Add to `.grype.yaml`: |
| 189 | +```yaml |
| 190 | +# Update requires [incompatibility]. [Severity] doesn't justify breaking changes. |
| 191 | +- vulnerability: GHSA-xxxx-xxxx-xxxx |
| 192 | + package: |
| 193 | + name: package.name/path |
| 194 | +``` |
| 195 | +
|
| 196 | +Commit: |
| 197 | +```bash |
| 198 | +git commit -s -m "$(cat <<'EOF' |
| 199 | +Ignore [package] CVEs incompatible with release-X.Y |
| 200 | + |
| 201 | +[Package] CVEs require versions with [incompatible dependency] |
| 202 | +incompatible with this branch's [current dependency]. |
| 203 | +EOF |
| 204 | +)" |
| 205 | +``` |
| 206 | + |
| 207 | +##### 10. Final Verification |
| 208 | + |
| 209 | +After fixing all packages: |
| 210 | + |
| 211 | +```bash |
| 212 | +# Verify no vulnerabilities remain |
| 213 | +grype . --config .grype.yaml -o table |
| 214 | + |
| 215 | +# Run build tests if not done in step 7 |
| 216 | +make unit |
| 217 | + |
| 218 | +# Review commits |
| 219 | +git log origin/release-0.X..HEAD |
| 220 | + |
| 221 | +# Verify no unexpected changes |
| 222 | +git diff |
| 223 | +``` |
| 224 | + |
| 225 | +Expected: "No vulnerabilities found", tests pass, commits follow @.agents/commit-templates.md, and no output from diff. |
| 226 | + |
| 227 | +##### 11. Create Pull Request (Optional) |
| 228 | + |
| 229 | +Agent generates ready-to-run PR command. User reviews commits, then copies and runs command if desired: |
| 230 | + |
| 231 | +```bash |
| 232 | +# Auto-generate gh pr create command |
| 233 | +CURRENT_BRANCH=$(git rev-parse --abbrev-ref HEAD) |
| 234 | +BASE_BRANCH=$(echo $CURRENT_BRANCH | sed 's/fix-\([0-9.]*\)-.*/release-\1/') |
| 235 | +COMMIT_COUNT=$(git log ${BASE_BRANCH}..HEAD --oneline | wc -l) |
| 236 | +PLURAL=$([[ $COMMIT_COUNT -eq 1 ]] && echo "" || echo "s") |
| 237 | +CVE_PLURAL=$([[ $COMMIT_COUNT -eq 1 ]] && echo "CVE" || echo "CVEs") |
| 238 | + |
| 239 | +# Find fork remote (non-submariner-io remote) |
| 240 | +FORK_REMOTE=$(git remote -v | grep -v 'submariner-io' | grep '(push)' | head -1 | awk '{print $1}') |
| 241 | +FORK_USER=$(git remote get-url ${FORK_REMOTE} 2>/dev/null | sed 's/.*github.com[:/]\([^/]*\).*/\1/') |
| 242 | + |
| 243 | +cat <<EOF |
| 244 | +
|
| 245 | +━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ |
| 246 | +Copy and run these commands to create PR: |
| 247 | +━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ |
| 248 | +
|
| 249 | +git push ${FORK_REMOTE} ${CURRENT_BRANCH} |
| 250 | +
|
| 251 | +gh pr create \\ |
| 252 | + --title "Fix ${CVE_PLURAL} in ${BASE_BRANCH}" \\ |
| 253 | + --body "See commit message${PLURAL} for details." \\ |
| 254 | + --base "${BASE_BRANCH}" \\ |
| 255 | + --head "${FORK_USER}:${CURRENT_BRANCH}" \\ |
| 256 | + --assignee "@me" |
| 257 | +
|
| 258 | +━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ |
| 259 | +EOF |
| 260 | +``` |
| 261 | + |
| 262 | +##### Common Issues |
| 263 | + |
| 264 | +- **CVE persists after update**: Verify you used correct FIXED-IN version; replace directives should have been caught in Step 3 |
| 265 | +- **New CVE appears after fix**: Dependency downgrades can introduce CVEs; fix immediately (Step 8 rescan catches these) |
| 266 | +- **Tests fail**: Try different version; check CI logs |
| 267 | +- **make unit "no route to host"**: User must run `sudo systemctl restart docker` (agent can't use sudo) |
0 commit comments