When Security Scanning Breaks CI: Fix the Risk, Not Just the Pipeline

At some point, almost every backend pipeline reaches this moment:

build: passed
tests: passed
security scan: failed

I hit this with Trivy in CI.

The application itself was working. The failure came from a vulnerability scan in the delivery pipeline.

The first instinct in this situation is dangerous because the fastest way to make the pipeline green is often not the safest way to fix the problem.

You can:

lower the severity threshold
ignore the CVE
allow the job to fail
remove the scanner
pin an old scan database

All of those can make CI stop blocking you.

None of them necessarily reduce the actual risk.

That experience changed how I approach security scan failures.

A failing scanner is a finding, not a diagnosis

A tool such as Trivy can tell you that a vulnerable package exists in an image or dependency graph.

That is useful.

But the raw result does not automatically tell you:

• whether your code uses the vulnerable functionality
• whether the package is direct or transitive
• whether the vulnerable package exists only in a build stage
• whether a patched version is available
• whether the finding comes from the base operating-system image
• whether upgrading it breaks another dependency

The scan answers:

What potentially vulnerable software is present?

Engineering still has to answer:

What is the right remediation for this system?

First, identify where the package came from

Before changing application code, I want to know the ownership path.

A vulnerability can enter through several layers:

application dependency
    ↓
transitive dependency
 
or
 
base container image
    ↓
OS package
 
or
 
build tooling

Those require different fixes.

If a Python library directly depends on the vulnerable package, upgrading the application dependency may be the right fix.

If the CVE belongs to an Ubuntu or Alpine package inherited from the base image, changing requirements.txt does nothing.

If the vulnerable package exists only in the build stage of a multi-stage image, the runtime exposure may be different again.

So I start with provenance before remediation.

"Upgrade everything" is not a good remediation strategy

It is tempting to run a broad dependency update and see whether the scanner becomes green.

That can create a second problem: now you have changed many packages without knowing which change actually fixed the vulnerability.

I prefer the smallest understandable remediation.

For application dependencies:

identify vulnerable package
    ↓
find dependency chain
    ↓
find minimum patched version
    ↓
check compatibility
    ↓
update lock file
    ↓
run tests
    ↓
rescan

This gives the change a reason.

The pull request can explain exactly what vulnerability was addressed and which dependency upgrade removed it.

Base images are part of your application

Container images make it easy to forget how much software comes from this one line:

FROM python:3.x

That tag is not just Python.

It includes an operating system and its packages.

A vulnerability in the image may have nothing to do with my FastAPI code, but it still ships with the service.

Sometimes the right fix is simply moving from an old base image digest/tag to a newer patched image.

That made me treat base images like dependencies rather than static infrastructure.

I want them versioned, reviewed, and refreshed deliberately.

Multi-stage builds reduce unnecessary exposure

Another useful lesson is that the runtime image should contain only what the application actually needs to run.

A build container may need:

compiler
headers
git
package managers
build tools

The final service often does not.

A multi-stage image can separate them:

FROM ... AS builder
# install build dependencies
# build packages
 
FROM ... AS runtime
# copy only runtime artifacts

This improves image size, but security scanning made another benefit more obvious: fewer runtime packages mean fewer vulnerability findings and a smaller attack surface.

The best scanner result is not suppressing 40 packages.

It is never shipping the unnecessary 40 packages in the first place.

Not every finding has an immediate patch

This is where security scanning becomes less mechanical.

Sometimes the scanner reports a high-severity vulnerability and there is no patched version available yet.

A real workflow needs a way to handle that without choosing between "ship nothing" and "ignore security."

For those cases I want an explicit exception with context:

CVE / advisory
affected component
why it is currently accepted
exposure/reachability assessment
mitigation if any
owner
review/expiry date

The expiry date matters.

A permanent ignore entry with no context is where temporary security decisions go to disappear.

Severity alone is not enough

CVSS or scanner severity is useful for prioritization, but it is not the whole risk model.

Two HIGH findings can mean very different things.

One may be in a network-facing parser used on every request.

Another may be in a package that exists in the image but is never executed by the application.

I do not use that as an excuse to dismiss findings.

I use it to decide remediation priority and whether a temporary exception is defensible.

The security gate should force a decision, not replace engineering judgment.

CI should prevent regressions, not create alert fatigue

If a repository has hundreds of known findings and every pipeline prints all of them, developers eventually stop reading the report.

That is a bad security system even if the scanner technically runs every time.

A more useful gate is designed around actionable change.

For example:

block newly introduced critical/high findings
track accepted existing findings
fail expired exceptions
regularly rescan the full image

The exact policy depends on the organization, but the principle is important:

A security gate should make new risk difficult to introduce and old risk difficult to forget.

It should not train developers to ignore a permanently red wall of output.

Run the same scanner before pushing

One practical annoyance with CI-only security scanning is feedback time.

If I can build a Docker image locally, I should be able to run roughly the same security check locally too.

That shortens the loop:

change dependency
    ↓
build image
    ↓
scan locally
    ↓
fix
    ↓
push

instead of:

change
push
wait for CI
scan fails
open logs
change again
push again

This is one reason I became interested in pre-check tooling around CI and security scans. The earlier a deterministic failure is visible, the cheaper it is to fix.

Security tools need policy around them

Installing Trivy is easy.

The harder work is deciding what its result means to your delivery process.

A useful policy answers:

Which severities block deployment?
Do we scan filesystem, dependencies, images, or all of them?
How are exceptions documented?
Who owns remediation?
When do exceptions expire?
How are base images refreshed?
Do preview/dev images use the same gate as production?

Without those decisions, developers end up improvising when the pipeline turns red.

And improvisation under delivery pressure usually optimizes for making the pipeline green.

What I learned

The important lesson from a security scan failure was not how to configure Trivy.

It was how to separate three things that are easy to mix together:

finding
risk
pipeline policy

The scanner produces a finding.

Engineers assess and remediate the risk.

CI enforces the policy the team has chosen around those risks.

When a security scan blocks my pipeline now, I try to work through it in this order:

• find where the vulnerable package came from
• determine whether a patched version exists
• make the smallest safe upgrade
• check base images separately from application dependencies
• reduce unnecessary runtime packages
• test after remediation
• rescan the final artifact
• document temporary exceptions with expiry dates
• avoid weakening the global gate just to fix one build

A green pipeline is useful.

A green pipeline achieved by hiding the finding is not the same thing as a safer service.