All your Traefik are belong to us

TL;DR

Tired of port juggling and certificate headaches? Run Traefik as a permanent service on your dev box. Point all your projects at it, and let them self-register. One wildcard certificate, real domains, no more tearing services down to free up port 443. Scales to production too. And then some.

Introduction

I run a lot of web development projects. Nowadays browsers sorta expect everything to be HTTPS so certificate management also is a thing. If you have an .dev tld, Google (who owns/governs .dev) even forces you to have it, even on local development environments. The problem? Every project wants to have something on port 443. Tearing down and starting up services becomes a day job.

I played with Traefik for a couple of projects, it made setting up a development box easy. Traefik deals with the certificates and I deal with.. developing. I took it a step further. Traefik as a service on my development box. Development projects as self registering entities, all with the same wildcard certificate.

Traefik

Traefik runs as a separate docker service controlled by systemd. Before we deal with how that is done, I’ll explain how Traefik normally is included in a project. In a normal project you define Traefik and what ports it exposes to the host system:

services:
  traefik:
    image: traefik:v3.0
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./etc:/etc/traefik-config
      - ./etc/traefik.yml:/etc/traefik/traefik.yml

You configure Traefik:

logs:
  level: debug

api:
  insecure: true

providers:
  file:
    directory: /etc/traefik-config
    watch: true
  docker:
    exposedbydefault: false

entrypoints:
  web-secure:
    address: :443
  plain-http:
    address: :80
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https

The important bit here is that we don’t expose all containers by default to Traefik. Instead we apply labels to containers we want to expose to Traefik and thus the host-system:

services:
  # [snip]
  web:
    image: nginx:latest
    volumes:
      - ./:/var/www
      - ./docker/nginx/conf.d/:/etc/nginx/conf.d:ro
    expose:
      - 80
    environment:
      - NGINX_HOST=localhost
    depends_on:
      php-fpm:
        condition: service_started
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.web.tls=true"
      - "traefik.http.routers.web.rule=Host(`somehost.localhost`)"
      - "traefik.http.services.web.loadbalancer.server.port=80"

You’ll notice that we don’t expose port 443 to Traefik. This is because we terminate SSL traffic on Traefik.

You need to create your certificates:

mkcert -install
mkcert myproject.localhost \*.myproject.localhost
mv myproject.localhost+1-key.pem etc/myproject.localhost.key
mv myproject.localhost+1.pem etc/myproject.localhost.pem

And tell Traefik where to find them, via the certificates.yml file:

---
tls:
  certificates:
    - certFile: /etc/traefik-config/myproject.localhost.pem
      keyFile: /etc/traefik-config/myproject.localhost.key

So this is local Traefik, now.. Let’s move it to a system service.

Traefik as a Service

Now we are going to use Traefik system wide. We start again with a compose file for Traefik to start it up as a service. The goal of Traefik was to eliminate multiple daemons taking ownership of port 443. In addition, we also want to do the very same thing for services as Postgres. And to finalize this setup, we also include some non-dockerized applications. The Traefik service looks almost identical to our configuration in a per project setup, the biggest differences are that we now use an external docker network and we include extra_hosts to allow non-dockerized applications to be used via Traefik.

services:
  traefik:
    image: traefik:v3.6
    container_name: traefik
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
      - "5432:5432"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./config:/etc/traefik/dynamic:ro
      - ./traefik.yml:/etc/traefik/traefik.yml:ro
      - "/opt/traefik-gateway/certs:/certs:ro"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    networks:
      - traefik

networks:
  traefik:
    external: true

We setup systemd to stop/start the service and we use the systemd reload infrastructure to rebuild the Traefik service.

[Unit]
Description=Traefik Gateway
After=docker.service
Requires=docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/opt/traefik-gateway
ExecStart=/usr/bin/docker compose up -d --remove-orphans
ExecStop=/usr/bin/docker compose down
ExecReload=/usr/bin/docker compose up -d --force-recreate --build --remove-orphans
TimeoutStartSec=0
TimeoutStopSec=120
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

The traefik.yml and certificates.yml don’t really change.

Projects!

Now that we have our infrastructure in place, we can have our projects run on the Traefik network. While I want to have everything exposed over my own Traefik, we don’t need the old configuration anymore. We can keep the projects configuration, and override it via docker-compose.override.yml by using some clever tricks of Docker specific YAML wizardry, and using !reset and !override. The former is to completely wipe a section from the configuration and the latter is to override the service. It will allow you to bypass any merge rule and completely override the section. See https://docs.docker.com/reference/compose-file/merge/#reset-value for more.

1. We don’t need a Traefik service anymore: !reset.
2. We override the labels of any Traefik bound service: !override
3. We add the external Traefik network to the config

Our docker-compose.override.yml looks like this:

services:
  traefik: !reset
  web:
    networks:
      - default
      - traefik
    labels: !override
      - "traefik.enable=true"
      - "traefik.docker.network=traefik"
      - "traefik.http.routers.project.rule=Host(`project.local.xyz.tld`)"
      - "traefik.http.routers.project.entrypoints=web,websecure"
      - "traefik.http.routers.project.tls=true"
      - "traefik.http.services.project.loadbalancer.server.port=80"

networks:
  traefik:
    external: true

We now have converted our former Traefik enabled project to our new infrastructure while maintaining backward compatibility with others who don’t run this setup. \o/

.env files

This blog is locally also exposed on the network. I employ the following trick to tell both Hugo and Traefik what the domain of the blog internally. In a .env file I added

BLOG_DOMAIN=blog.local.mydomain.tld

In the docker-compose.yml file I added this:

services:
  hugo:
    command:
      - server
      - "--bind"
      - "0.0.0.0"
      - "--baseURL=https://$BLOG_DOMAIN"
      - "--buildDrafts"
      - "--buildFuture"
      - "--disableFastRender"
      - "--appendPort=false"
    # [snip]
    labels:
      - "traefik.http.routers.blog.rule=Host(`$BLOG_DOMAIN`)"

And because Hugo serves things on port 1313 by default we map this port to 80 via this label:

- "traefik.http.services.blog.loadbalancer.server.port=1313"

Postgres

Traefik is not just for HTTP(s) services, you can also expose Postgres via Traefik. You’ll need SNI for this to work and it took a little effort to get postgres to work, but it works flawlessly now.

Unfortunately the Postgres images are maintained by the people of Docker hub and they don’t want to expose include_dir or include_if_exists so you need to jump through some hoops to get your custom config to work. I took the stock configuration of postgres and added include_dir = '/etc/postgresql/conf.d'. To later remove it and just add the missing pieces directly into the regular config.

And I changed the entrypoint to be mine so I can chown and chmod files correctly:

#!/bin/sh
set -eu

DIR=/etc/postgresql
CONFD=$DIR/conf.d
CERTS=$DIR/certs
mkdir -p $CONFD $CERTS

cp /mnt/postgresql.conf $DIR/postgresql.conf
cp -rp /mnt/postgres.conf.d/* $CONFD
cp -rp /mnt/postgres.certs/* $CERTS

chmod 644 $DIR/postgresql.conf $CONFD/* $CERTS/*
chmod 600 $CERTS/*key*.pem
chown -R postgres:postgres $DIR

exec docker-entrypoint.sh $*

And my additional (ssl) config looks a bit like this:

ssl = on
ssl_cert_file = '/etc/postgresql/certs/myproject.local.xyz.tld.pem'
ssl_key_file = '/etc/postgresql/certs/myproject.local.xyz.tld-key.pem'

And when we combine this all our docker-compose section looks like this:

services:
  # [snip]
  db:
    image: postgres:15
    expose:
      - 5432
    command:
      - postgres
      - "-c"
      - "config_file=/etc/postgresql/postgresql.conf"
    # [snip]
    entrypoint:
      - /entrypoint.sh
    networks:
      - default
      - traefik
    volumes:
      - dbdata:/var/lib/postgresql/data
      - ./db:/var/db
      - ./docker/postgres/entrypoint.sh:/entrypoint.sh
      - ./docker/postgres/postgresql.conf:/mnt/postgresql.conf
      - ./docker/postgres/conf.d:/mnt/postgres.conf.d
      - ./docker/certs:/mnt/postgres.certs
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=traefik"
      - "traefik.tcp.routers.newerproject-pg.rule=HostSNI(`myproject.local.xyz.tld`)"
      - "traefik.tcp.routers.newerproject-pg.entrypoints=dbsecure"
      - "traefik.tcp.routers.newerproject-pg.tls.passthrough=true"
      - "traefik.tcp.services.newerproject-pg.loadbalancer.server.port=5432"

And now connecting to postgres works like this:

psql "sslmode=require host=myproject.local.xyz.tld dbname=myproject" \
    -U myproject

The only thing that I couldn’t get to work with SNI was pgloader , but I’m sure that was me making a mistake.

The infrastructure

My machines are provisioned via Ansible, and this Traefik project wouldn’t be complete if it weren’t in Ansible. I had to change my Ansible a bit to make it all happen.

Docker

In the past my development machine (via the role desktop-gui) had a docker recipe. I moved that recipe out of the desktop-gui role into its own role. The idea is, that this Traefik setup is also a real deployment method for services I offer to paying customers. Therefore I had to deploy docker without relying on my desktop role. The role itself is very simple. I deploy a deb822 sources file with GPG keys of Docker and I ensure it gets installed and the service gets enabled.

Configuring Traefik in Ansible

We tell Traefik which SSL certificates to use, and if we use a dynamic config.

Traefik owns the certs in /opt/traefik-gateway/certs

cd /opt/traefik-gateway/certs
mkcert local.xyz.tld "*.local.xyz.tld"

The dynamic config is the section where it states traefik_http_services.

traefik:
  hosts:
    quasar:
      traefik_dynamic_config_enabled: false
      traefik_tls_certificates:
      - cert: local.xyz.tld.pem
        key: local.xyz.tld.key
    sputnik-odin:
      traefik_tls_certificates:
      - cert: local.xyz.tld.pem
        key: local.xyz.tld.key
      traefik_dynamic_config_enabled: true
      traefik_http_services:
        small-server:
          hostname: small.local.xyz.tld
          port: 9001
        blog:
          hostname: blog.local.xyz.tld
          port: 1400

And finally my traefik.yml looks like this:

providers:
  docker:
    exposedByDefault: false
  file:
    directory: /etc/traefik/dynamic
    watch: true

entryPoints:
  web:
    address: ":80"
  websecure:
    address: ":443"
  dbsecure:
    address: ":5432"

PiHole

In order to allow every host in the network to access my development projects I needed some kind of DNS service. So one of the first things I added was a pihole docker service, this allowed me to configure my local domain somewhere without the need for a full-fledged DNS server. This pihole service is also served via Traefik:

services:
  pihole:
    image: pihole/pihole:latest
    container_name: pihole
    restart: unless-stopped
    dns:
      - "8.8.8.8"
    ports:
      - "53:53/tcp"
      - "53:53/udp"
    expose:
      - 80
    environment:
      TZ: "America/Curacao"
      FTLCONF_webserver_api_password: ""
      FTLCONF_dns_upstreams: "8.8.8.8"
      FTLCONF_dns_listeningMode: "ALL"
      FTLCONF_misc_etc_dnsmasq_d: "true"
    volumes:
      - pidata:/etc/pihole
      - dnsmasqdata:/etc/dnsmasq.d
      - ./dnsmasq/02-local-dns.conf:/etc/dnsmasq.d/02-local-dns.conf
    extra_hosts:
      - "host.docker.internal:host-gateway"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.pihole.tls=true"
      - "traefik.docker.network=traefik"
      - "traefik.http.routers.pihole.rule=Host(`pihole.local.xyz.tld`)"
      - "traefik.http.routers.pihole.entrypoints=websecure"
      - "traefik.http.services.pihole.loadbalancer.server.port=80"
    networks:
      - traefik

volumes:
  pidata: null
  dnsmasqdata: null

networks:
  traefik:
    external: true

Pihole is configured to expose local DNS:

pihole:
  hosts:
    quasar:
      pihole_exposed_ports:
        - "53:53/tcp"
        - "53:53/udp"
      pihole_local_dns_records:
        - name: router.local.xyz.tld
          ip:  192.168.0.1

Let’s encrypt

I’m currently developing another project to fully support DNS01 for Let’s encrypt so all my services run with a proper certificate instead of self signed ones. This is mostly to support mobile app development as you cannot easily drop self-signed certs on Android or iPhone. Which is a must have for my company.

My local SaaS

PiHole wasn’t the only service I wanted system-wide. I had more candidates that deserved to be a service for all the projects to have.

Mailhog vs Mailpit

Any project using e-mail in the past had MailHog . Because Mailhog seems dead, it hasn’t received any TLC in the past couple of years, I converted it to MailPit .
The deployment looks very similar to the pihole and the traefik roles and exposes itself on the network. Every project can now send e-mail to this service. Which saves me having to configure it for each project. And instead of Mailpit you could also install MailCrab . I chose Mailpit because of a coin flip.

Postgres

I’m also considering maybe running Postgres as a service, perhaps even running multiple versions so I can easily test various postgres deployments. But this is still up in the air.
Between writing the article and publication I actually had a legitimate cause to give in to this idea. So Postgres now a service.

Future plans

I also expect to add Redis as a service which would then allow for most of my projects to fully utilize the shared infrastructure and it would potentially mean I can run a docker-stack on less powerful machines because most of the additional services are hosted on a more powerful machine.

Caddy

Caddy is technically a per-project service, but I wanted to give it a small shoutout because it allows me to run web applications without either Apache or Nginx. I think especially on development machines this deserves some attention.

I find that it works better than for example nginx, as nginx refuses to start if a backend is down, which means you need to wait instead of just starting up and let the backend service to come up and start serving files.

Caddy just serves static files and exposes FastCGI middleware:

services:
  web:
    build:
      context: .
      dockerfile: ./docker/caddy/Dockerfile
    # [snip]
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=traefik"
      - "traefik.http.routers.newerproject.rule=Host(`newerproject.local.xyz.tld`)"
      - "traefik.http.routers.newerproject.entrypoints=websecure"
      - "traefik.http.routers.newerproject.tls=true"
      - "traefik.http.services.newerproject.loadbalancer.server.port=80"
      - "traefik.http.middlewares.https-forward.headers.customrequestheaders.X-Forwarded-Proto=https"
      - "traefik.http.routers.newerproject.middlewares=https-forward"

My Caddyfile looks like this:

:80 {
  root * /var/www/public
  php_fastcgi php:9000
  file_server
}

I might see a future where I run a global Caddy service, but for now it stays local to individual projects.

In conclusion

By spending a couple of evenings on getting Traefik to run, enable PiHole, Mailpit, and using SNI for Postgres I’ve built a development environment that can be plugged into a box and use a real production server. The same concepts apply for hosting multiple projects and multiple clients.

The plan was to make development easy, and as a side-effect I made something that scales well to production. I think this setup earns its place next to any Kubernetes setup. Smaller in scale, and equally powerful (yes, I said what I said). Small shops where Kubernetes is overkill might be able to use this pattern effectively.