HomeLab

Why I Built This

Most of my projects are things I build. This is the one I have to live with.

I started with the usual homelab motivation: learn some Linux, run some services, stop paying for cloud instances that sit there doing nothing.

Then the problems started.

A DNS setup that worked at home didn't work remotely. A Tailscale route black-holed traffic. A torrent client could reach UDP trackers with a test probe but couldn't actually announce itself. A monitoring tool was reporting completely ridiculous CPU numbers. A backup could finish successfully without me knowing whether the data could actually be restored.

None of these were particularly exotic problems. They were just problems that don't show up when the only thing running is a demo.

So I started keeping incident notes, writing actual postmortems, building runbooks, and putting operational state somewhere I could see it. That's probably been the most useful part of the project.

The Two Hosts

OptiPlex 3060 Micro

The OptiPlex does the heavy lifting. It's an i5-8500T with integrated graphics, running AlmaLinux 10 and 19 rootless Podman containers.

I deliberately went with AlmaLinux instead of whatever distro happened to be easiest. It's RHEL-compatible, so the things I'm learning here — systemd, SELinux, firewalld, storage, service management — are directly useful for the kind of Linux infrastructure work I want to do.

The machine has a small SSD and a ~1 TB HDD, so storage layout matters more than it would on a modern server with a pile of NVMe.

Raspberry Pi 4

The Pi handles the stuff I want available even when the main machine isn't:

• DNS and filtering
• monitoring
• dashboard
• operational utilities

It's also on battery backup. That makes it a surprisingly good little infrastructure node. If the OptiPlex goes down, I still want DNS and monitoring to exist long enough to tell me why.

The Pi isn't powerful. That's kind of the point.

Storage: The Disk Dictates the Layout

The server has roughly 119 GB of SSD and 932 GB of HDD.

The HDD is an SMR drive, which is perfectly reasonable for large sequential files and considerably less exciting when you start throwing lots of random writes at it.

So I stopped treating storage as "put the big stuff on the big disk." Instead, I treat it according to how it gets written.

The SSD gets:

• PostgreSQL
• thumbnails and generated data
• container storage
• service configuration
• transcode cache
• other latency-sensitive state

The HDD gets:

• photos
• videos
• media
• other large, mostly-sequential data

The difference is noticeable when you get the layout wrong.

The HDD is mounted by UUID with nofail and a systemd device timeout. Services that depend on it use RequiresMountsFor= so they cannot happily start writing into an empty directory on / when the disk isn't mounted.

That particular failure mode is nasty because everything appears to work until you eventually wonder where all your root filesystem space went.

Why Everything Is a systemd Unit

I originally just wanted containers. Then I realised I also wanted them to behave like services.

All 19 containers run rootless Podman and are defined as Quadlets. systemd handles startup, dependencies, restarts, logging, and ordering.

No docker compose up sitting around as the thing keeping the system alive. No rootful container daemon. Linger means the services start without requiring me to log in first.

It's a pretty boring setup. That's exactly what I wanted.

Digest-pinned images

I don't use floating image tags. Every image is pinned to a sha256 digest, so restarting a service doesn't accidentally turn into an upgrade.

When I want to upgrade something, I:

• read the release notes;
• compare the new configuration with what I'm running;
• pull the new image;
• update the digest;
• restart that service.

It's more manual than latest. That's intentional.

SELinux Without Turning It Off

SELinux was one of the things I specifically wanted to get comfortable with.

The easy homelab answer to a container permission problem is usually to add :Z. Or, worse, to disable SELinux. I wanted to actually understand what was happening.

The large data volume gets the appropriate container_file_t label once, so containers can use it without repeatedly relabelling hundreds of gigabytes. Paths under the home directory follow a different rule and use :Z where appropriate.

It's not complicated once the model makes sense. The useful part was learning to stop treating SELinux errors as something to bypass and start treating them as information about which boundary I had crossed.

Rootless Doesn't Mean "No Permissions Problems"

One slightly unintuitive part of the setup was figuring out how UID mapping actually behaves.

With rootless Podman, container root is mapped to the unprivileged host user. So a container running as UID 0 isn't suddenly root on the host. That makes UID 0 inside some containers useful for keeping ownership consistent.

Other applications insist on running as a particular non-root UID. Those use an explicit UserNS=keep-id mapping where necessary.

The important part wasn't memorising the configuration. It was finally understanding which UID exists where.

One Name, Two Networks

Every service gets a simple internal name: jellyfin.home, photos.home, dash.home.

At home, those names need to resolve to the LAN address. Away from home, my devices need the Tailscale address.

I initially tried solving this with subnet routing. That was a mistake. I advertised the LAN through Tailscale, and managed to create a route that looked perfectly reasonable while silently black-holing traffic because the router wasn't forwarding it.

So instead of making routing more complicated, I stopped routing. I use split-horizon DNS.

                    jellyfin.home
                         │
              ┌──────────┴──────────┐
              │                     │
          LAN client             Tailscale client
              │                     │
        AdGuard Home            CoreDNS
              │                     │
        LAN address           Tailnet address
              │                     │
              └──────────┬──────────┘
                         │
                       Caddy
                         │
                    19 services

LAN devices don't need a VPN. My devices can use the same hostname from anywhere on the tailnet. And because the CoreDNS resolver is only reachable through Tailscale, LAN clients never accidentally receive the tailnet address.

This was one of those cases where the eventual solution was simpler than the first one.

Why it's plain HTTP

The services aren't public. They're internal .home names, behind the LAN/Tailscale boundary, and I don't need a public certificate for them.

I could set up a private CA and distribute trust to every device in the house. I could also explain to my housemate why their browser is complaining about a certificate. Or I could use HTTP.

I chose HTTP. Not every homelab problem needs a more complicated solution just because production systems normally use HTTPS.

Backups: The Part I Actually Care About

Every night, roughly 24 GB of important data gets copied offsite and encrypted. Every week, the backup gets tested by actually restoring files.

The remote uses a copy-based approach rather than mirroring. If I accidentally delete something locally, the deletion doesn't immediately become a deletion remotely. Previous versions are kept in timestamped backup directories.

It's less elegant than a perfect mirror. I'd rather have the ugly copy.

Verification

The remote is encrypted, which means I can't just compare the provider's ciphertext hash with my local plaintext. So verification happens in two layers:

• Cheap check: make sure everything exists and sizes look right.
• Expensive check: randomly restore 15 files, decrypt them, and compare their checksums with the originals.

The second one is the important test. If the restore works, I know the network worked, the remote data exists, the encryption configuration works, the keys work, and the restored bytes actually match what I started with.

The result becomes JSON and goes onto the dashboard as:

• OK — backup and restore verification passed
• WARN — expected differences, usually because local files changed after the last backup
• FAIL — something is actually missing or doesn't match

I don't want a dashboard that tells me everything is green because a script exited with code 0.

A stupidly useful optimisation

The first version walked the remote directory tree individually. It took around nine minutes. Using a bulk listing dropped that to roughly twenty seconds.

That's the kind of optimisation you don't think about until you have to stare at your own backup job every night.

Media isn't backed up. At my current upload speed, re-uploading media I can simply download again isn't worth the time it would take. Photos, configs, databases and other irreplaceable data are the priority.

Monitoring

The dashboard runs on the Pi and collects:

• host metrics
• container metrics
• backup status
• service health
• logs

The log viewer can tail all 19 rootless containers in one place.

And yes, the monitoring has lied to me. Podman's per-container CPU number is currently inflated by roughly 305× in the viewer. So when something says a container is using 3000% CPU, I don't immediately start panicking. It's documented in the runbook.

A monitoring system that confidently gives you bad information can be more dangerous than having no metric at all.

The Incident That Made the Project Worth Writing About

This is probably my favourite part of the whole homelab.

A large torrent sat at zero peers. Naturally, I blamed the ISP, CGNAT, UDP filtering, and Podman. All reasonable guesses. All wrong.

A raw UDP probe to the same trackers worked. Then I ran the same probe from inside the container. That worked too. So outbound UDP wasn't the problem.

The difference was the source socket.

The application was announcing from its own listening socket through rootless Podman's userspace port proxy. Replies couldn't make their way back through that path. The test probe, meanwhile, used an ephemeral source port and behaved completely normally.

Same network. Same container. Different socket. That was the actual bug.

Then there was another problem: LAN peer discovery uses multicast, and multicast wasn't crossing the Podman bridge. And the tracker couldn't help either because both clients were behind the same CGNAT address. So I had three different discovery mechanisms failing for three different reasons.

The fix was moving the service to host networking. The result was pretty dramatic:

• UDP trackers: 0 → 4–5 per torrent
• DHT: 0 → 122 connected nodes
• LAN transfer: ~5 MB/s → ~58 MB/s

And then the fix broke other things.

Three services and the reverse proxy had been referring to the container by its old container-network hostname. The host's LAN address also wasn't reachable from the container the way I expected because of the firewall path, so host.containers.internal was the correct target. Then Caddy's reload reported success while continuing to serve the old configuration. Only restarting it actually applied the change.

That incident taught me more about containers and networking than a lot of tutorials did.

Constraints

This thing is still a cheap homelab. That means it has limits.

Memory

The OptiPlex currently has 8 GB RAM, which is getting uncomfortable with the photo stack and 19 containers. A 16 GB upgrade is planned.

I also managed to learn the hard way that not every DDR4 SO-DIMM is compatible with this generation of Intel memory controllers. Apparently RAM can have lore.

CGNAT

There is no useful inbound IPv4 connectivity. That's annoying for peer-to-peer workloads, but it's also one reason the homelab has almost nothing exposed publicly. Remote administration goes through Tailscale.

SMR storage

The HDD is great for large media files. It is not where I want databases or workloads doing sustained random writes. So the storage layout is built around that limitation rather than pretending the disk is something it isn't.

What I Actually Learned

The biggest thing this project taught me is that building something and operating something are very different activities.

Building a service is usually: does it work?

Operating it becomes:

• What happens when the disk disappears?
• What happens when DNS breaks?
• How do I know the backup worked?
• What if the monitoring is wrong?
• Can somebody else use this without asking me how it works?
• What happens when the fix itself breaks something else?

I didn't set out to build some miniature enterprise platform. I wanted somewhere cheap to learn.

Somewhere along the way, I ended up learning that the boring parts — backups, logs, runbooks, service dependencies, failure modes, recovery — are where most of the actual engineering is.

And unlike most of my projects, this one keeps giving me bugs after I ship it. Which is probably why I've learned the most from it.