Keith RockhillSenior Network Engineer

Two clusters, one router, and no open ports.

The Rocknet Lab is where I practice the work I do on carrier networks, at a much smaller scale. I rebuilt it this fall, going from one three-node cluster to two: a production side for services that have to stay up, and a lab side for building and breaking things.

Status, November 4, 2026: the move is complete. This site now runs on the production cluster, on one of the HA pairs. Its partner node holds a ZFS replica that updates every two minutes, so if the active node fails, HA restarts the site there with at most about two minutes of changes lost. The original cluster has been wiped and rebuilt as the new lab. The first full night of backups ran clean, and the production backups copied over to the second drive. Still on my list: a failover drill on each pair, a pull test of the second WAN link and the offsite backup copy.

Internetpublic traffic Cloudflare edgefilters traffic before the tunnel outbound tunnel · no inbound ports EdgeRouter 12dual WAN failover · zone firewall · VLANs Production Cluster5 nodes · Proxmox VE · HA + ZFS replication Management nodePBS · PDM · monitoring backup drives Pair A Active Standby ZFS replevery 2 min HA restarts guests here Pair B Active Standby ZFS replevery 2 min HA restarts guests here Firewall Lab Cluster3 nodes · Proxmox VE · own network Build Test Try to break it New builds, configs and toolsstart here, walled off from production only tested workis promoted Offsite copy (planned)encrypted before upload Alerts (planned)push to my phone Rocknet Lab: Clusters and Network Drawn byKeith Rockhill Call signKB8YYE
Public traffic only reaches production through a Cloudflare tunnel. Production and lab sit on separate networks behind the EdgeRouter's firewall.

Production Cluster

Five Lenovo ThinkCentre Mini nodes (Intel i5, 32 GB RAM and a 1 TB SSD each) in one Proxmox VE cluster, installed on ZFS.

Lab Cluster

The three original Dell OptiPlex 9020s, wiped and rebuilt as their own Proxmox cluster. New builds, configs and tools start here. The lab has its own network: the firewall lets it reach the internet and send backups to the management node, and nothing else on the production side. It also runs the container I use to automate the builds, covered below.

Network

A Ubiquiti EdgeRouter 12 is the edge for both clusters.

Live: Router and Clusters

This is the new edge router and both clusters, live. Prometheus polls the router over SNMP every 30 seconds, and Proxmox Datacenter Manager supplies the combined cluster totals. A job on the management node publishes a cleaned copy of the numbers, and this web server fetches it. The page never talks to the router, the monitoring system or Datacenter Manager directly, and no addresses or machine names are shown.

Loading live data. This panel needs JavaScript.

Backups and Alerts

Proxmox Backup Server takes nightly backups of production to a 1 TB drive, keeping 7 daily, 4 weekly and 3 monthly copies. Lab backups go to a second drive, along with a copy of the production backups, so losing one drive doesn't lose everything. Planned next: an encrypted copy offsite and push alerts to my phone.

Hardware

The new Rocknet rack on casters: patch panels, network gear and five Lenovo ThinkCentre Mini nodes
The new rack, with the five ThinkCentre production nodes in the upper half.
Production
5× Lenovo ThinkCentre Mini, Intel i5, 32 GB, 1 TB SSD
Lab
3× Dell OptiPlex 9020
Edge router
Ubiquiti EdgeRouter 12, two SFP WAN links
Switching
2× TP-Link TL-SG105 (lab and spare)
Backup storage
USB enclosure with a 1 TB drive and a 256 GB SSD, plus a 931 GB drive
Software
Proxmox VE 9, Proxmox Backup Server, Proxmox Datacenter Manager

Building It with AI in the Loop

A container on the lab cluster runs Claude Code, and I use it as an agentic dev environment. I install Proxmox on each node by hand. A deploy script it helped me write does the rest in phases: cluster joins, HA rules, replication, backup storage, the management tools and the lab. Parts of it were tested in throwaway containers first. The first full run on the real hardware turned up bugs, and I fixed them as they came up.

I'm still the one doing the engineering. I have to know the Proxmox API, EdgeOS, Terraform and nginx well enough to say what to build, review what comes back and catch it when it's wrong. What changes is speed. Once I know what right looks like, I get there a lot faster.

Earlier Lab Projects

The Original Cluster

Three OptiPlex 9020s clustered as "Rocknet" behind a TP-Link ER605, with an NFS share on a USB drive for ISOs and backups. It hosted this site through a Cloudflare Tunnel, with no inbound ports open, until the move in November 2026. Those three machines are now the lab cluster.