GitHub runners¶

Linux jobs of a private repository run here instead of on GitHub's hosted runners, whose minutes a free organisation pays for. The Actions Runner Controller is two components: arc-controller runs the controller and its CRDs, arc-runners one scale set per repository. A job gets a fresh runner pod that exits when the job ends. macOS jobs cannot run here and stay on GitHub.

At a glance¶


Scale sets	`homelab` and `homelab-wait` on `somuch-cloud/extractor.cafe`
Workflow label	`runs-on: homelab`
Depends on	OpenBao for the GitHub App in `kv/arc-runners/github-app`
If it is down	Jobs on `runs-on: homelab` queue on GitHub until a runner registers; nothing else notices
Health check	`kubectl -n arc-systems get pods` → the controller and one listener per scale set Running
Namespaces	`arc-systems` for the controller and the listeners, `arc-runners` for the runner pods, `arc-jobs` for the job pods the container hook starts
Files	`payload/platform/arc-controller/`, `payload/platform/arc-runners/`, and the three namespaces' files under `payload/platform/security/`

Configuration¶

Setting	Why
Controller and scale set chart on the same `targetRevision`	The controller deletes an `AutoscalingRunnerSet` whose chart version differs from its own, and Argo CD recreates it, in a loop; Renovate moves both in one PR
`githubConfigUrl` on the repository, not the organisation	No other repository in the organisation can schedule code onto the cluster
`githubConfigSecret: github-app`	A GitHub App rather than a PAT: no expiry and a token scoped to the installed repository — see GitHub App
`controllerServiceAccount` spelled out	The chart finds the controller with Helm `lookup`, which Argo CD does not run
`minRunners` and `maxRunners`	No idle pods between pushes; the ceiling lets several pull requests run their parallel Linux jobs at once without queueing, while the requests stay a fraction of the workers' memory
`containerMode: kubernetes-novolume`	A job with `services:` or `container:` gets its own pod, started by the container hook through the API server. No privileged Docker daemon and no PVC per job. Where that pod runs, and with which token: Job pods
`ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER: "false"`	A job without `container:` runs in the runner pod itself, as on a hosted runner
`arc-runners` and `arc-jobs` enforce `baseline`	The runner image uses `sudo`, and a service container such as PostgreSQL starts as root; neither passes `restricted`
`arc-jobs` is exempt from the user-namespace kill	buildah builds the images as root without `CAP_SYS_ADMIN`, which `baseline` forbids, so it re-executes itself in a user namespace; Tetragon records those instead of killing them
The runner image as a full reference with a digest	Renovate tracks it; the chart passes the same image to the hook for the job pod's init container
`resources`	Unmeasured; the request fits a test suite with a database beside it, and the limit stops a runaway build before it starves the node
A second scale set, `homelab-wait`, for `ci-ok` alone	The wait holds its runner until every other check on the commit has finished; on the shared set, as many open pull requests as `maxRunners` fill it with waits and the checks they wait for never start. It runs no build, so it has no container mode and a small request

Job pods¶

The chart's own ServiceAccount for kubernetes-novolume may read every Secret in arc-runners, where the GitHub App key and each runner's JIT config live, and a step without container: runs in the runner pod with that token. The homelab set therefore runs its hook one namespace over.

Setting	Why
`serviceAccountName: homelab-runner`	The chart then renders no Role of its own; `job-pods-rbac.yaml` grants the ServiceAccount rights in `arc-jobs` only
`ACTIONS_RUNNER_KUBERNETES_NAMESPACE: arc-jobs`	The hook creates the job pod and its environment Secret there, so the Secret verbs it needs reach nothing else
The Role is the hook's `requiredPermissions`, Secret `get` and `list` included	The hook checks every verb before each command and fails the job if one is missing; a step can still read the environment of another job running at the same time, from the same repository
`default` ServiceAccount in `arc-jobs` without a token	A job pod never calls the API server

All jobs share arc-jobs and one Role, so a job can reach the pods and Secrets of any job running beside it. That is accepted because no repository with other contributors will use these runners; if that changes, it takes one scale set and namespace per trust level, each with its own Role.

Controller¶

The arc-controller values, and why:

Setting	Why
`flags.watchSingleNamespace: arc-runners`	The controller gets a Role in this namespace instead of a ClusterRole over every namespace
`logLevel: info`, `logFormat: json`	The chart's default is `debug`, which logs every reconcile; JSON is what Loki parses
Restricted security context on the controller and the listener	Both are distroless and run as non-root; the namespace enforces `baseline` and audits `restricted`

Metrics¶

metrics in the controller values turns on /metrics on port 8080 of the controller and of every listener it starts. The listener's are the scale set's view from GitHub (assigned jobs, desired, idle and registered runners, job start-up and run times), the controller's the runner pods it manages. Both PodMonitors in payload/platform/arc-controller/podmonitors.yaml set honorLabels, so the metrics keep their own namespace, the runners' namespace, instead of the pod's arc-systems.

The GitHub Actions Runners dashboard in payload/platform/arc-controller/grafana-dashboards.yaml reads them. The job metrics carry no scale set label, so its job panels take the scale set from the listener pod's name. Upstream's sample dashboard is not vendored: it filters on actions_github_com_scale_set_name, which this version does not export, and most of its panels stay empty.

Network policy¶

Every rule in arc-runners.yaml and arc-jobs.yaml under payload/platform/security/network-policies/. Unlike the other namespaces, egress is every public address rather than a list of names.

Rule	Why
Ingress from the namespace and the node only	Runners and job pods serve nothing; the listener in `arc-systems` reaches GitHub, not the runners, and a runner reaches its job pod through the API server
Egress to the API server, `homelab` runner pods only	The container hook creates the job pod and execs each step into it; the `homelab-wait` runners run no hook, and job pods carry no token
Egress to `0.0.0.0/0` on 80 and 443, minus the private, carrier-grade and link-local ranges	A job downloads whatever its workflow names (actions, toolchains, packages), which no name list keeps up with. Cutting the private ranges keeps the LAN, the nodes and every in-cluster service out of reach

The controller's and listeners' egress, GitHub's API and the Actions service only, and Prometheus' scrape of their metrics port, are in payload/platform/security/network-policies/arc-systems.yaml.

GitHub App¶

One app per organisation, installed on each repository that has a scale set.

Create the app under the organisation's Settings → Developer settings → GitHub Apps → New GitHub App: any name and homepage URL, Webhook → Active unticked, Repository permissions → Administration: Read and write, Metadata: Read-only, installable on this account only.
On the app's page, note the App ID and Generate a private key; the .pem file downloads.
Install App → Only select repositories → the repository. The installation ID is the number at the end of the installation's URL.
Store all three:

make bao-secrets    # answers the arc-runners/github-app prompts

Push the secret through and wait for the listener:

kubectl -n arc-runners annotate externalsecret github-app force-sync="$(date +%s)" --overwrite
kubectl -n arc-systems get pods -l app.kubernetes.io/component=runner-scale-set-listener

The scale set then appears under the repository's Settings → Actions → Runners.

Usage¶

A job opts in with its label:

runs-on: homelab

Job shape	What runs where
Steps only	In the runner pod, on the runner image
`container:`	A job pod from that image; steps exec into it
`services:`	Needs a `container:` too; the services join the job pod and answer on `localhost`

The runner image is GitHub's minimal actions-runner, not the hosted ubuntu-latest: git, curl, jq, unzip and sudo, and no language toolchains. A job installs what it needs, pinned, or uses a setup-* action.

Adding a scale set¶

Install the GitHub App on the repository.
In payload/platform/arc-runners/application.yaml, copy a chart source with its own helm.releaseName, githubConfigUrl and runnerScaleSetName; a set that runs job pods keeps containerMode, serviceAccountName and ACTIONS_RUNNER_KUBERNETES_NAMESPACE.
In job-pods-rbac.yaml, a ServiceAccount, Role and binding for the set's hook, or homelab-runner again when the jobs share its trust level, see Job pods.
In payload/platform/security/network-policies/arc-runners.yaml, the container-hook policy selects actions.github.com/scale-set-name homelab only; a set that runs a hook needs the same rule for its name.
A repository that runs ci-ok needs a -wait twin of the set, like homelab-wait, for the reason in Configuration.
Workflows opt in with runs-on: <runnerScaleSetName>.

Health check¶

kubectl -n arc-runners get autoscalingrunnerset,ephemeralrunner
kubectl -n arc-systems get pods
kubectl -n arc-systems logs -l app.kubernetes.io/component=runner-scale-set-listener --tail=50

A job waiting on runs-on: homelab with no runner pod in arc-runners means the listener is not getting it; its log and the controller's say why.

Pitfalls¶

Job Container is required. A job with services: and no container:. The hook runs services only beside a job container.
No Docker daemon. The image carries the docker CLI, but nothing answers it: docker build and docker run steps fail.
No listener until the app is in OpenBao. An empty kv/arc-runners/github-app syncs cleanly; the controller then fails to register the scale set with GitHub and says so in its log.
Removing a scale set takes the controller. Its Roles carry the actions.github.com/cleanup-protection finalizer, which only the controller removes; take the scale set out first and let it go before touching arc-controller.