Docker Registry with TLS

Manually importing every image onto every cluster node is quickly unworkable. The solution: a private registry hosted inside the cluster, accessible over HTTPS with a self-signed certificate and protected by a password.

Installing Docker

Docker is installed on cube01 to build and push multi-architecture images from the local network.

sudo apt-get install ca-certificates curl gnupg lsb-release
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/debian $(lsb_release -cs) stable" \
  | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin

Daemon configuration (/etc/docker/daemon.json):

{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "experimental": true,
  "log-driver": "json-file",
  "storage-driver": "overlay2",
  "log-opts": {
    "max-size": "100m"
  }
}

sudo systemctl enable docker
sudo systemctl start docker

# Multi-architecture support (arm64 + amd64)
sudo apt-get install binfmt-support qemu-user-static
docker buildx create --use --platform=linux/arm64,linux/amd64 --name multi-platform-builder
docker buildx inspect --bootstrap

# Add current user to the docker group
sudo groupadd docker
sudo usermod -aG docker $USER

Initial HTTP Registry (first step)

The registry is first deployed without TLS to validate the basic setup. Kubernetes manifests create a dedicated namespace, a PVC, a Deployment, a Service, and an Ingress.

On each node, add the registry to /etc/hosts:

<IP-cube04>    registry    docker-registry.local

Then create /etc/rancher/k3s/registries.yaml so containerd knows about the mirror:

mirrors:
  docker-registry:
    endpoint:
      - "http://docker-registry.local:80"

Validation test:

curl http://docker-registry.local:80/v2/_catalog
# Returns the list of repositories

Switching to TLS with a Local CA

.cluster is not a public domain, so Let’s Encrypt cannot issue a certificate. Instead, an internal CA is created and used to self-sign the registry certificate.

Generating the CA and Certificate

mkdir registry-ca && cd registry-ca

# CA key and certificate (valid 10 years)
openssl genrsa -out ca.key 4096
openssl req -x509 -new -nodes \
  -key ca.key -sha256 -days 3650 \
  -out ca.crt -subj "/CN=Cluster Registry CA"

# Key and CSR for registry.cluster
openssl genrsa -out registry.key 4096
openssl req -new -key registry.key -out registry.csr -subj "/CN=registry.cluster"

# SAN extension
echo "subjectAltName = DNS:registry.cluster" > registry.ext

# Sign the certificate
openssl x509 -req \
  -in registry.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out registry.crt -days 3650 -sha256 -extfile registry.ext

Trusting the CA on All Machines

On the workstation and on every Raspberry Pi node:

sudo cp ca.crt /usr/local/share/ca-certificates/cluster-ca.crt
sudo update-ca-certificates
# On nodes only:
sudo systemctl restart containerd

Kubernetes Resources

Namespace and TLS secret

kubectl create namespace registry
kubectl create secret tls registry-tls \
  --cert=registry.crt --key=registry.key -n registry

Basic auth

sudo apt install apache2-utils
htpasswd -Bc htpasswd admin
kubectl create secret generic registry-auth \
  --from-file=htpasswd -n registry

PVC (Longhorn storage)

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: registry-pvc
  namespace: registry
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 20Gi
  storageClassName: longhorn

Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: registry
  namespace: registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app: registry
  template:
    metadata:
      labels:
        app: registry
    spec:
      containers:
        - name: registry
          image: registry:2
          ports:
            - containerPort: 5000
          env:
            - name: REGISTRY_HTTP_ADDR
              value: "0.0.0.0:5000"
            - name: REGISTRY_AUTH
              value: "htpasswd"
            - name: REGISTRY_AUTH_HTPASSWD_REALM
              value: "Registry Realm"
            - name: REGISTRY_AUTH_HTPASSWD_PATH
              value: "/auth/htpasswd"
          volumeMounts:
            - name: storage
              mountPath: /var/lib/registry
            - name: auth
              mountPath: /auth
      volumes:
        - name: storage
          persistentVolumeClaim:
            claimName: registry-pvc
        - name: auth
          secret:
            secretName: registry-auth

Service

apiVersion: v1
kind: Service
metadata:
  name: registry
  namespace: registry
spec:
  selector:
    app: registry
  ports:
    - port: 5000
      targetPort: 5000

Traefik IngressRoute (HTTPS)

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: registry
  namespace: registry
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`registry.cluster`)
      kind: Rule
      services:
        - name: registry
          port: 5000
  tls:
    secretName: registry-tls

Configuring k3s on All Nodes

Edit /etc/rancher/k3s/registries.yaml on each node:

mirrors:
  registry.cluster:
    endpoint:
      - "https://registry.cluster"

configs:
  registry.cluster:
    auth:
      username: admin
      password: <password>
    tls:
      ca_file: /usr/local/share/ca-certificates/cluster-ca.crt

# On the control plane
sudo systemctl restart k3s
# On workers
sudo systemctl restart k3s-agent

Testing

# Login
docker login registry.cluster

# Tag and push
docker tag nginx registry.cluster/nginx:test
docker push registry.cluster/nginx:test

# Check via the API
curl -u admin:<password> https://registry.cluster/v2/_catalog

A {"errors":[{"code":"UNAUTHORIZED"...}]} response without credentials confirms that TLS is working and the registry is properly protected.

In Kubernetes manifests, images are referenced directly:

image: registry.cluster/myapp:latest

k3s resolves the hostname and pulls the image from the private registry using the configured credentials.