Trivy Operator¶

Aqua Security's Trivy Operator scans everything the cluster runs and writes every result back as a Kubernetes object. Reading those objects is Vulnerability Triage; this page is the operator itself.

At a glance¶

The eks- and rke2- compliance specs are for other distributions and are left out rather than reported as failing.

Configuration¶

The values in application.yaml that are not defaults:

Alerting¶

The rules in prometheusrule.yaml. Reports expire and are rebuilt every scannerReportTTL, so a "new" finding is one absent from the reports two days ago rather than from the previous scrape, and every rule waits longer than a rebuild.

The two "new" rules also require the operator to have been reporting two days ago, so a fresh cluster does not page on every existing finding.

TrivyContainerNotScanned compares every running container against the containers that have a VulnerabilityReport and names the difference:

count by (namespace, container) (
  kube_pod_container_info
  * on (namespace, pod) group_left ()
  (max by (namespace, pod) (kube_pod_status_phase{phase="Running"} == 1))
)
unless on (namespace, container)
count by (namespace, container) (
  label_replace(trivy_image_info, "container", "$1", "container_name", "(.+)")
)

• Keyed on (namespace, container), not the digest: containerd's platform digest and the registry's tag digest differ permanently for mirrored images.
• Restricted to Running pods, or every completed Job counts as unscanned.
• A for of several full sweeps, so the TTL churn does not page.

The dashboard's stat and by-severity panels take max by (image_digest) before summing, and workload panels join ReplicaSets against kube_replicaset_spec_replicas > 0, so a finding counts once per image and a superseded revision drops out. The "Worst workloads" table stays per container on purpose: it answers where a finding runs.

History¶

Reports live for scannerReportTTL and Prometheus for its retention, so neither can answer whether a count went up over a quarter. The findings-history CronJob in findings-history.yaml lists every report kind once a day and prints one JSON line per namespace, and per compliance framework, to stdout; Alloy ships it like any container log and Loki keeps that one stream far longer than the rest (retention_stream in loki/application.yaml). The dashboard's last row reads it. Images are counted once per digest per namespace, the same rule the dashboard applies.

{namespace="trivy-system", container="findings-history"} | json | kind="vulnerabilities" | namespace="argocd"

SBOM export¶

sbomGenerationEnabled makes the operator write one CycloneDX SbomReport per container next to the VulnerabilityReport. Nothing here reads them; Dependency-Track in the workloads repository uploads every one of them nightly and keeps the history Trivy's 24h reports cannot. sbom-readers.yaml is the ClusterRole and binding that let its upload job list them, and the ReplicaSets, so it can skip the reports of superseded revisions: the apps project may not create cluster-scoped RBAC, and which workload reads the operator's findings is decided here.

Report pruning¶

SbomReports and ExposedSecretReports carry no TTL, unlike the VulnerabilityReports: they are owned by the scanned ReplicaSet and live as long as it does, and Deployments keep ten superseded revisions scaled to zero. Those SBOMs were over half the reports' bytes in etcd, and each API server holds them decoded in its watch cache. The report-prune CronJob in report-prune.yaml deletes the two kinds for every ReplicaSet scaled to zero, daily before the Dependency-Track upload, which skips those reports anyway. The two ScanOnlyCurrentRevisions settings keep the operator from writing them again. Its log is one JSON line per kind with the count deleted.

Usage¶

The dashboard is a summary; the objects have the detail.

# What has been scanned, worst first
kubectl get vulnerabilityreports -A --sort-by='.report.summary.criticalCount'

# The findings themselves, for one workload
kubectl get vulnerabilityreports -n <ns> <name> -o json \
  | jq '.report.vulnerabilities[]
        | select(.severity=="CRITICAL")
        | {id: .vulnerabilityID, pkg: .resource,
           installed: .installedVersion, fixed: .fixedVersion}'

# Misconfiguration, RBAC and node checks
kubectl get configauditreports -A
kubectl get rbacassessmentreports,clusterrbacassessmentreports -A
kubectl get infraassessmentreports -A

# Which compliance controls are failing
kubectl get clustercompliancereports k8s-cis-1.23 -o json \
  | jq '.status.summary, (.status.summaryReport.controlCheck[]
        | select(.totalFail > 0))'

Health check¶

Reconcile running containers against scanned ones, keyed on (namespace, container) like the alert; the output is every running container with no VulnerabilityReport, and healthy is an empty output:

comm -23 \
  <(kubectl get pods -A --field-selector status.phase=Running -o json \
      | jq -r '.items[] | .metadata.namespace as $ns | .spec.containers[] | "\($ns) \(.name)"' | sort -u) \
  <(kubectl get vulnerabilityreports -A -o json \
      | jq -r '.items[] | "\(.metadata.namespace) \(.metadata.labels["trivy-operator.container.name"])"' | sort -u)

Pitfalls¶

Four ways a workload ends up with no VulnerabilityReport, all silent from the outside:

A completed scan job counts against concurrentScanJobsLimit until scanJobTTL deletes it, so a backlog drains in batches of the limit, the chart's default, once per TTL; that is the expected shape, not a stall.