Trivy Operator¶

Aqua Security's Trivy Operator scans everything the cluster runs and writes every result back as a Kubernetes object. Reading those objects is Vulnerability Triage; this page is the operator itself.

At a glance¶

The eks- and rke2- compliance specs are for other distributions and are left out rather than reported as failing.

Configuration¶

The values in application.yaml that are not defaults:

Alerting¶

Five rules in prometheusrule.yaml. Reports expire and are rebuilt every 24h, so a "new" finding is one absent from the reports two days ago rather than from the previous scrape, and every rule waits longer than a rebuild.

The two "new" rules also require the operator to have been reporting two days ago, so a fresh cluster does not page on every existing finding.

TrivyContainerNotScanned compares every running container against the containers that have a VulnerabilityReport and names the difference:

count by (namespace, container) (
  kube_pod_container_info
  * on (namespace, pod) group_left ()
  (max by (namespace, pod) (kube_pod_status_phase{phase="Running"} == 1))
)
unless on (namespace, container)
count by (namespace, container) (
  label_replace(trivy_image_info, "container", "$1", "container_name", "(.+)")
)

• Keyed on (namespace, container), not the digest: containerd's platform digest and the registry's tag digest differ permanently for mirrored images.
• Restricted to Running pods, or every completed Job counts as unscanned.
• for: 2h, several times a full sweep, so the 24h TTL churn does not page.

The dashboard's stat and by-severity panels take max by (image_digest) before summing, and workload panels join ReplicaSets against kube_replicaset_spec_replicas > 0, so a finding counts once per image and a superseded revision drops out. The "Worst workloads" table stays per container on purpose: it answers where a finding runs.

History¶

Reports live 24h and Prometheus keeps ten days, so neither can answer whether a count went up over a quarter. The findings-history CronJob in findings-history.yaml lists every report kind once a day and prints one JSON line per namespace, and per compliance framework, to stdout; Alloy ships it like any container log and Loki keeps that one stream for a year (retention_stream in loki/application.yaml). The dashboard's last row reads it. Images are counted once per digest per namespace, the same rule the dashboard applies.

{namespace="trivy-system", container="findings-history"} | json | kind="vulnerabilities" | namespace="argocd"

Usage¶

The dashboard is a summary; the objects have the detail.

# What has been scanned, worst first
kubectl get vulnerabilityreports -A --sort-by='.report.summary.criticalCount'

# The findings themselves, for one workload
kubectl get vulnerabilityreports -n <ns> <name> -o json \
  | jq '.report.vulnerabilities[]
        | select(.severity=="CRITICAL")
        | {id: .vulnerabilityID, pkg: .resource,
           installed: .installedVersion, fixed: .fixedVersion}'

# Misconfiguration, RBAC and node checks
kubectl get configauditreports -A
kubectl get rbacassessmentreports,clusterrbacassessmentreports -A
kubectl get infraassessmentreports -A

# Which compliance controls are failing
kubectl get clustercompliancereports k8s-cis-1.23 -o json \
  | jq '.status.summary, (.status.summaryReport.controlCheck[]
        | select(.totalFail > 0))'

Health check¶

Reconcile running images against scanned ones; treat a difference as a missing scan until proven otherwise:

kubectl get pods -A -o json \
  | jq -r '.items[].spec.containers[].image' | sort -u | wc -l
kubectl get vulnerabilityreports -A -o json \
  | jq -r '.items[].report.artifact.repository' | sort -u | wc -l

Pitfalls¶

Four ways a workload ends up with no VulnerabilityReport, all silent from the outside:

A completed scan job counts against concurrentScanJobsLimit until scanJobTTL deletes it, so a backlog drains in batches of ten every ten minutes; that is the expected shape, not a stall.