DinD Forgejo Runner on Kubernetes
Why Switch
I recently received an email from Fedora regarding the switch from Pagure to Forgejo. Being quite out of the loop, I hadn’t heard of the plans to do so, or even of Forgejo. In my homelab, I was running Gitea with a runner on Kubernetes using my own Helm chart. This worked well and I had no compliants in functionality. Yet, that email stirred things up, planting a groundless idea that I should make the same switch. Forgejo’s self-comparison to Gitea provided the justification; namely, Forgejo originated as a fork of Gitea after a for-profit company silently took over. This smells like the beginning of a freemium model similar to GitLab. I have no inherit issues with freemium software and could have stayed on Gitea until they started adding price tags to previously free features but, the takeover seems a bit hostile from the outside. That, alongside Forgejo’s commitments to the community and security, I decided to try it out.
Deployment on Kubernetes
Forgejo provides an official Helm chart. Gitea does as well, but I had not used it (for some forgotten reason). As such, I can’t speak to the differences and how seamless the switch between these charts might be.
This time around, I opted to use Forgejo’s chart. Since this will only support 1-2 users, sqlite is more than fine but enabling WAL with gitea.config.database.SQLITE_JOURNAL_MODE=WAL helped out, especially considering this is running on spinning rust, rather than SSDs. Again, twoqueue will be performant enough but gitea.config.cache.HOST='{"size":100, "recent_ratio":0.25, "ghost_ratio":0.5}' will improve efficiency. Both of these are documented in Forgejo’s recommendations. In order to later migrate repositories from my local Gitea to Forgejo, I also had to set gitea.config.migrations=ALLOW_LOCALNETWORKS=true. The chart doesn’t include a network policy, so that is specified in extraDeploy. For reference, here is my full values.yaml:
clusterDomain: kube.lukedavidson.org
strategy:
type: Recreate
containerSecurityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
add:
- SYS_CHROOT
privileged: false
readOnlyRootFilesystem: true
runAsGroup: 1000
runAsNonRoot: true
runAsUser: 1000
service:
http:
type: ClusterIP
port: 3000
clusterIP: None
ssh:
type: LoadBalancer
port: 2424
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: traefik
traefik.ingress.kubernetes.io/router.entrypoints: websecure
hosts:
- host: git.app.lukedavidson.org
paths:
- path: /
pathType: Prefix
port: http
resources:
limits:
memory: 2Gi
requests:
memory: 1Gi
replicaCount: 1
persistence:
enabled: true
size: 32Gi
storageClass: openebs-zfspv
annotations:
helm.sh/resource-policy: keep
gitea:
admin:
existingSecret: prod-forgejo-admin
email: admin-email-here
passwordMode: keepUpdated
metrics:
enabled: true
serviceMonitor:
enabled: true
config:
server:
SSH_PORT: 2424
SSH_LISTEN_PORT: 2424
ROOT_URL: https://git.app.lukedavidson.org
DISABLE_REGISTRATION: true
database:
DB_TYPE: sqlite3
SQLITE_JOURNAL_MODE: WAL
cache:
ADAPTER: twoqueue
HOST: '{"size":100, "recent_ratio":0.25, "ghost_ratio":0.5}'
migrations:
ALLOW_LOCALNETWORKS: true
extraDeploy:
- apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
name: prod-forgejo-admin
namespace: forgejo
spec:
encryptedData:
password: encrypted-password-here
username: encrypted-username-here
template:
metadata:
name: prod-forgejo-admin
namespace: forgejo
type: Opaque
- apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: prod-forgjo
spec:
podSelector: {}
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
app.kubernetes.io/name: traefik
app.kubernetes.io/instance: traefik-kube-system
ports:
- port: 3000
protocol: TCP
- ports:
- port: 2424
protocol: TCP
Making it Run(ners)
My Gitea setup had a Docker-in-Docker runner which I used exclusively for building and pushing containers to Gitea’s built-in package system. This was a must, but unfortunately Forgejo does not currently provide a Helm chart for it. There is an open issue for a more Kubernetes-native runner, but I used their example docker-compose runner as a starting point. Unlike the Gitea runner, the Forgejo runner requires an init container to register. That docker compose example provides an init container, but I have a few issues with it. For reference, here is the example init container script:
bash -ec '
while : ; do
forgejo-runner create-runner-file --connect --instance http://forgejo:3000 --name runner --secret {SHARED_SECRET} && break ;
sleep 1 ;
done ;
sed -i -e "s|\"labels\": null|\"labels\": [\"docker-cli:docker://code.forgejo.org/oci/docker:cli\",\"node-bookworm:docker://code.forgejo.org/oci/node:20-bookworm\"]|" .runner ;
forgejo-runner generate-config > config.yml ;
sed -i -e "s| level: info| level: debug|" config.yml ;
sed -i -e "s|network: .*|network: host|" config.yml ;
sed -i -e "s|^ envs:$$| envs:\n DOCKER_HOST: tcp://docker:2376\n DOCKER_TLS_VERIFY: 1\n DOCKER_CERT_PATH: /certs/client|" config.yml ;
sed -i -e "s|^ options:| options: -v /certs/client:/certs/client|" config.yml ;
sed -i -e "s| valid_volumes: \[\]$$| valid_volumes:\n - /certs/client|" config.yml ;
chown -R 1000:1000 /data
'
First, the runner will unconditionally re-register, creating a brand new runner with every container restart. Another issue I encountered was my own doing, but was easy enough to fix in the init container. I was running the forgejo-runner as non-root and the volume permissions were incorrect (I forgot to add securityContext.fsGroup=1000 on the Deployment spec), so the runner would successfully register, but then fail to write /data/.runner in the container. This caused it to continually re-register, leading to 400+ runners registered to my Forgjo instance in seconds. Both of my fixes are included in the init container command here:
if [[ -f /data/.runner ]]; then exit 0; fi # exit if already registered
if [[ ! -w /data ]]; then exit 1; fi # fail if we can't write /data/.runner
forgejo-runner register --no-interactive --token $(RUNNER_SECRET) --name $(RUNNER_NAME)
Note, I also removed the config generation and sed edits from the init script, opting to use a ConfigMap instead.
Here is the full Deployment spec:
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "forgejo-runner.fullname" . }}
labels:
{{- include "forgejo-runner.labels" . | nindent 4 }}
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
{{- include "forgejo-runner.selectorLabels" . | nindent 6 }}
template:
metadata:
labels:
{{- include "forgejo-runner.labels" . | nindent 8 }}
spec:
securityContext:
fsGroup: 1000
automountServiceAccountToken: false
initContainers:
- name: runner-register
image: {{ .Values.runner.image.repository }}:{{ .Values.runner.image.tag }}
command:
- /bin/bash
- -c
args:
- |
if [[ -f /data/.runner ]]; then exit 0; fi # exit if already registered
if [[ ! -w /data ]]; then exit 1; fi # fail if we can't write /data/.runner
forgejo-runner register --no-interactive --token $(RUNNER_SECRET) --name $(RUNNER_NAME) --instance $(FORGEJO_INSTANCE_URL) || exit $?
env:
- name: RUNNER_NAME
value: prod-forgejo-runner
- name: RUNNER_SECRET
valueFrom:
secretKeyRef:
name: prod-runner-secret
key: token
- name: FORGEJO_INSTANCE_URL
value: https://git.app.lukedavidson.org
resources:
limits:
cpu: '0.5'
ephemeral-storage: 100Mi
memory: 64Mi
requests:
cpu: 100m
ephemeral-storage: '0'
memory: 64Mi
volumeMounts:
- name: runner-data
mountPath: /data
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
privileged: false
readOnlyRootFilesystem: true
runAsUser: 1000
runAsGroup: 1000
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
containers:
- name: daemon
image: {{ .Values.docker.image.repository }}:{{ .Values.docker.image.tag }}
env:
- name: DOCKER_TLS_CERTDIR
value: /certs
resources:
limits:
cpu: '1'
ephemeral-storage: 3Gi
memory: 4Gi
requests:
cpu: 100m
ephemeral-storage: '0'
memory: 64Mi
securityContext:
privileged: true
volumeMounts:
- name: docker-certs
mountPath: /certs
- image: {{ .Values.runner.image.repository }}:{{ .Values.runner.image.tag }}
name: runner
command:
- /bin/bash
- -c
args:
- |
until nc -z localhost 2376; do
echo 'waiting for docker daemon...'
sleep 5
done
forgejo-runner --config config.yml daemon
env:
- name: DOCKER_HOST
value: tcp://localhost:2376
- name: DOCKER_CERT_PATH
value: /certs/client
- name: DOCKER_TLS_VERIFY
value: '1'
resources:
limits:
cpu: '1'
ephemeral-storage: 3Gi
memory: 4Gi
requests:
cpu: 100m
ephemeral-storage: '0'
memory: 64Mi
volumeMounts:
- name: docker-certs
mountPath: /certs
readOnly: true
- name: runner-data
mountPath: /data
- name: runner-config
mountPath: /data/config.yml
subPath: config.yml
readOnly: true
- name: tmp
mountPath: /tmp
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
privileged: false
readOnlyRootFilesystem: true
runAsUser: 1000
runAsGroup: 1000
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
volumes:
- name: docker-certs
emptyDir: {}
{{- if .Values.persistence.enabled }}
- name: runner-data
persistentVolumeClaim:
claimName: {{ include "forgejo-runner.fullname" . }}
{{- else }}
- name: runner-data
emptyDir: {}
{{- end }}
- name: runner-config
configMap:
name: {{ include "forgejo-runner.fullname" . }}
- name: tmp
emptyDir: {}
And the ConfigMap:
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "forgejo-runner.fullname" . }}
labels:
{{- include "forgejo-runner.labels" . | nindent 4 }}
data:
config.yml: |
log:
level: info
job_level: info
runner:
file: .runner
capacity: 1
envs:
DOCKER_HOST: tcp://localhost:2376
DOCKER_TLS_VERIFY: 1
DOCKER_CERT_PATH: /certs/client
timeout: 1h
insecure: false
fetch_timeout: 5s
fetch_interval: 2s
report_interval: 1s
labels: ["docker-build:docker://git.app.lukedavidson.org/homelab/docker-build:29.1.5"]
cache:
enabled: true
port: 0
dir: ""
external_server: ""
secret: ""
host: ""
proxy_port: 0
actions_cache_url_override: ""
container:
network: host
enable_ipv6: false
privileged: false
options: -v /certs/client:/certs/client:ro
workdir_parent:
valid_volumes:
- /certs/client
docker_host: "-"
force_pull: false
force_rebuild: false
host:
workdir_parent:
If you plan to use the runner to build docker images, take attention to labels: ["docker-build:docker://git.app.lukedavidson.org/homelab/docker-build:29.1.5"] in the config. By default, the image used by Forgejo actions jobs doesn’t include docker tooling. Since I only plan to use this runner to build containers, I set the image to one of my own, which includes the docker cli. If you prefer otherwise, you can set the container.image per-job in each runner job spec. For more infomration on using docker in Forgejo actions, see their documentation. Here is the Dockerfile for the aforementioned image:
ARG BASE_REPO=docker.io/library/docker
ARG BASE_TAG=cli
FROM ${BASE_REPO}:${BASE_TAG}
RUN apk add --no-cache \
git \
curl \
bash \
nodejs \
npm
This image will allow steps like actions/checkout@v4, docker/login-action@v3, and docker/build-push-action@v5 to work in your job. If you really want it, here is also the runner job spec to build this container:
name: docker-build build
on:
push:
branches: [ main ]
paths:
- 'containers/docker-build/**'
jobs:
build:
runs-on: docker-build
env:
PACKAGE: docker-build
DOCKER_CONTEXT: containers/docker-build
steps:
- uses: actions/checkout@v4
- name: Resolve build variables
id: vars
run: |
base_image=$(cat $DOCKER_CONTEXT/BASE_IMAGE)
base_repo=${base_image%%:*}
base_tag=${base_image##*:}
registry="${{ github.server_url }}"
registry="${registry#https://}"
registry="${registry#http://}"
repo="$registry/${{ github.repository_owner }}/$PACKAGE"
tag=$base_tag
echo "::group::Resolved build variables"
echo "PACKAGE = $PACKAGE"
echo "DOCKER_CONTEXT = $DOCKER_CONTEXT"
echo "base_image = $base_image"
echo "base_repo = $base_repo"
echo "base_tag = $base_tag"
echo "registry = $registry"
echo "repo = $repo"
echo "tag = $tag"
echo "::endgroup::"
echo "base_repo=$base_repo" >> $GITHUB_OUTPUT
echo "base_tag=$base_tag" >> $GITHUB_OUTPUT
echo "registry=$registry" >> $GITHUB_OUTPUT
echo "repo=$repo" >> $GITHUB_OUTPUT
echo "tag=$tag" >> $GITHUB_OUTPUT
- name: Login to registry
uses: docker/login-action@v3
with:
registry: ${{ steps.vars.outputs.registry }}
username: ${{ secrets.ACTIONS_USER }}
password: ${{ secrets.ACTIONS_TOKEN }}
- name: Build and push image
uses: docker/build-push-action@v5
with:
context: ${{ env.DOCKER_CONTEXT }}
push: true
build-args: |
BASE_REPO=${{ steps.vars.outputs.base_repo }}
BASE_TAG=${{ steps.vars.outputs.base_tag }}
tags: |
${{ steps.vars.outputs.repo }}:${{ steps.vars.outputs.tag }}
${{ steps.vars.outputs.repo }}:latest
The last piece is the BASE_IMAGE file next to the Dockerfile, which contains docker.io/library/docker:29.1.5-cli and used during the build to specify which image version to base off of.
Migrating Repos
If you are running an older version of Gitea, before Forgejo became a hard fork, you could have upgraded from Gitea to Forgejo in place and avoided all of this new seutp. Since I didn’t have that route, the final piece was to move the data over. Once Forgejo is set up, you can create a new migration and pull directly from Gitea. This was extremely easy and only took a few minutes. Just remember the Forgejo config needs ALLOW_LOCALNETWORKS if Gitea is running locally.
All in all, there isn’t a very noticiable difference in Forgejo functionality yet, but I am happy with the switch.