Recent Posts

Proxmox On-Site and Off-Site Backup

Nov 25, 2025 • Proxmox, backup, ZFS, storage

Failure resilience keeps a service running through a failure. Backup gets your data back after one. It’s worth being clear about which of those you’re buying, because a mirrored pool feels like a backup right up until the moment you need one.

A ZFS mirror protects you from a drive dying. It does absolutely nothing about rm -rf in the wrong directory, which it will faithfully replicate to both drives at full speed.

So the useful question isn’t “am I backed up?” but “which failures does each copy of my data actually cover?”

What On-Site Backup Is For

Two things, and they’re worth naming separately because they have different recovery shapes.

Fat-finger recovery. Somebody — probably you — deletes the wrong file, breaks a config, or upgrades something that turns out to be a mistake. This is by far the most common reason to reach for a backup, and it wants frequent snapshots and a fast, low-ceremony restore. You’ll use this one dozens of times for every time you use the others.

Local storage failure beyond what redundancy covers. A mirror survives one drive. A RAIDZ1 pool survives one drive. Lose two in a RAIDZ1, or one in a stripe, and the pool is gone regardless of how good ZFS is. Drives bought at the same time from the same batch have an unhelpful habit of failing at similar times, which makes this less theoretical than it sounds.

On-site backup answers both, and it answers them fast, because the data is sitting on the same network.

What Off-Site Backup Is For

Also two things, and both of them are the ones you never get to practise.

Local storage failure combined with local backup failure. The pool fails, you reach for the on-site backup, and discover that it was on the same shelf, the same UPS, or the same controller.

Site disaster. Fire, flood, theft, a burst pipe above the rack. Everything in the building is gone at once, and the only copies that matter are the ones that weren’t in it.

Off-site is slow to restore from and you will hardly ever touch it. That’s fine. It isn’t for convenience, it’s for the day the building is a write-off.

The Baseline

Proxmox Backup Server does the on-site job well. Backups are incremental and deduplicated, so keeping many restore points costs far less space than keeping many full copies, and it can verify what it’s holding rather than assuming. It can also sync a datastore to a second PBS elsewhere, which is the cleanest route to off-site.

My baseline is a ZFS mirror on non-ECC RAM for both the nodes and the backup server, with every node dumping to PBS regardless of what tier its guests are in. Even a guest with no resilience at all gets backed up, because backup is the one thing that covers the mistake you’re actually going to make.

It’s tempting to argue that a backup server which already replicates off-site doesn’t need storage redundancy of its own — if the pool dies, the off-site copy is still there. That’s true, and I’d still mirror it. A backup server that loses its pool leaves you restoring your fat-finger mistakes across a slow link from the off-site copy, which turns a five-minute annoyance into an afternoon. The mirror isn’t protecting the data, it’s protecting the fast path to it.

Where the Amount of Data Changes Everything

This is the part that makes backup planning messier than it first looks.

Some services carry almost no data. A DNS resolver, an NTP server, a small internal web app: a configuration file, a package list, some logs. These are ideal candidates for everything. ZFS replication between nodes is nearly free, so they can run HA and fail over in seconds. Backups are small, so you can keep many restore points and push them off-site over an ordinary internet connection without thinking about it.

Other services churn large files continuously. A DVR is the clearest example: hours of new video every day, most of it never watched. Every option above gets expensive at once.

• Replication would have to move gigabytes constantly, saturating the drives and the network to protect data that mostly isn’t worth protecting.
• HA is therefore off the table, which is fine — nobody is harmed by a recorder being down for an hour.
• Off-site backup would need an upload budget that no residential link is going to provide, and would burn it on footage with a shelf life measured in days.

The mistake is applying one policy to both. If you treat the DVR like the DNS server, you’ll spend a fortune and still miss recordings. If you treat the DNS server like the DVR, you’ll take a fifteen-minute outage for something that could have failed over in seconds.

That last cell is the one worth dwelling on. “Off-site the whole DVR” is unaffordable, but “off-site nothing from it” is usually wrong too. There’s normally a small, precious subset — the recordings you actually kept, the config that took an afternoon to get right — and separating that from the bulk at the storage layout level is what makes the decision cheap.

Test the Restore

A backup you have never restored from is a hypothesis.

Restoring a guest to a different node, or to a scratch VM, takes a few minutes and tells you whether the thing you’ve been diligently running every night actually produces a working service. It also tells you how long a real restore takes, which is the number you’ll want when something is broken and someone asks when it’ll be back.

I’d rather find out on a Tuesday afternoon than during the disaster.

Back to Resilience

The dividing line here — how much data a service carries — is the same one that decides which resilience tier it belongs in. Small-data services get HA and generous backups because both are cheap. Large-data services get neither, and instead get a clear-eyed decision about which parts of their data are actually worth keeping.

Work that out once per service, write it down, and most of the hard questions answer themselves.

Entry-Level Failure Resilience with Proxmox VE

Nov 24, 2025 • Proxmox, ZFS, storage, networking, hardware

If Murphy’s Law tells us that anything that can go wrong will go wrong, then planning failure resilience means working through anything that can go wrong. For each kind of failure, the question I keep coming back to is a simple one: how long is the service outage it creates, and what does shortening that outage cost?

I adopted virtualization to modernize and simplify the administration of my network services. Focusing on failure resilience is perhaps less common in a homelab, but it’s what I actually cared about. The values that drive a high-end homelab overlap substantially with a small IT shop or an SMB: a handful of services that people notice immediately when they stop, and nobody on call at 3 a.m.

This post is the entry-level version. It is not about building something impressive. It is about the smallest configuration that turns each failure from an evening of work into a few seconds of nothing.

Backups are the other half of this and get their own post. Resilience keeps a service running through a failure. Backup gets your data back after one. They are different jobs, and it is worth being clear which one you are buying.

Hardware Baseline

I’m done with spinning drives, noisy fans, and high power consumption 24x7. So I’ve stayed with compact, low-TDP CPUs and M.2 NVMe storage.

At the low end you’re likely to spend more on storage than on compute no matter what you do, so I think it’s worth paying a bit more for name-brand drives with good warranties and reliability records. I’ve stayed with the Samsung 990 line, but beware of the confusing naming.

There are three: the 990 EVO, the 990 EVO Plus, and the 990 PRO, each costing a bit more than the last. The PRO is the one with a dedicated DRAM cache, and I don’t see that adding much value behind the large DRAM requirements of ZFS. The slightly higher performance of the EVO Plus likely makes no difference either, given the limited CPU power of low-TDP parts and the network interface bottleneck in front of them. So I’d stay with the plain 990 EVO unless you need the 4 TB density, which the EVO tops out below.

Storage Layout

ZFS is clearly the right choice for local storage here. Ceph is a possibility for distributed storage, but it’s overkill for a homelab and it wants more nodes, more network, and more attention than this scale justifies.

Always mirror the boot device. A Proxmox node whose boot drive fails is a node that is simply gone until you rebuild it, and rebuilding it is an evening, not a minute. A two-drive ZFS mirror turns that into a drive swap you can do whenever you feel like it.

It is tempting to stripe instead, because striping two drives gets you the capacity of both. Resist it.

Failure Modes

Here is the whole exercise: list what can break, work out the outage, then decide what shortening it is worth.

Boot Device Failure

The node can’t boot because its boot device has failed.

Without mitigation, the outage runs until you have reinstalled Proxmox and restored the guests, which realistically means hours and a lot of attention.

Configure the boot device as a ZFS mirror across two drives. The node keeps running on the surviving drive and the outage becomes zero. Replacing the failed drive is scheduled work rather than emergency work.

Network Link Failure

The node’s hardware is fine and the switch is fine, but the physical link between them has failed. A cable, a port, a transceiver.

Without mitigation, everything on that node is unreachable until someone physically visits it.

Configure bonded interfaces using LACP across multiple physical links to the same switch. Failover is sub-second and nobody notices.

Network Switch Failure

The node’s hardware is fine, but the switch it connects to has failed. LACP to a single switch does nothing for you here — the whole bond is behind the thing that broke.

Configure multiple interfaces with MLAG using links to two different switches. This is the point where the cost starts to climb, because it needs a second switch and a pair that supports MLAG between them. It is also the first mitigation on this list I would consider genuinely optional for a homelab.

Compute Failure

The physical hardware underlying a node fails in some way that isn’t storage: loss of power, a kernel panic, a CPU that bursts into flames.

This is the one that HA is for, and it is where entry-level gets interesting.

What High Availability Actually Requires

Proxmox VE will fail a guest over to another node automatically, but only inside a cluster with quorum.

The second thing to understand is where the guest’s disk lives. Automatic failover needs the guest’s storage to be available on the node taking over. Without shared storage, that means ZFS replication: Proxmox periodically sends snapshots of the guest’s disk to the other nodes so a copy is already there when it’s needed.

Replication runs on a schedule, and that schedule is your data loss window. Replicate every fifteen minutes and a node failure costs you up to fifteen minutes of that guest’s writes. Replicate every minute and you pay for it in constant disk and network traffic. Nothing about HA makes this go away — it is the price of not having shared storage, and it is the number to decide deliberately rather than accept by default.

So the honest summary of automatic failover is: the service comes back in seconds, on a slightly older copy of its data.

Putting It Together

Not everything deserves the same treatment. I find it useful to sort guests into tiers and be explicit about what each one buys.

A few things are true across all of them. The node boots to run the guest, and the guest boots to serve clients, so both need to survive a drive failure. Some guests also touch bulk storage that is far too large to replicate. And every node dumps to a backup server regardless of tier, because none of the above protects you from deleting the wrong thing.

Where This Runs Out

The tiers above are really a proxy for one question: how much data does the guest carry?

A DNS resolver or an NTP server holds a configuration file and not much else. Replicating it every few minutes is nearly free, so it can sit in Tier 1 and fail over in seconds.

A DVR churning large files is the opposite case. Replicating it continuously would saturate the network and the drives to protect data that is, frankly, replaceable. It belongs in Tier 3 with a sensible backup and no HA at all.

That dividing line — how much data a service carries — turns out to decide almost everything about both its resilience and its backups, which is where the backup post picks up.

Installing Proxmox on Asustor Flashstor 6 Gen 2 FS6806X

Nov 22, 2025 • Proxmox, hardware, networking, storage

I bought the Asustor FS6806X due to its appealing features for a home lab NAS and virtualization host.

1. Six hot-swappable M.2 NVMe drive bays
2. Up to 64 GB of ECC RAM
3. Built-in 10 GbE NIC

I discovered significant challenges while installing Proxmox on the FS6806X:

1. The built-in 10 GbE NIC is not supported by the Proxmox kernel.
2. Without an HDMI port, there’s no way to interact with the Proxmox installer directly.

I worked around these, but it’s not for the faint of heart.

Add a Video Card for Setup

1. Get an M.2 to PCI-e riser cable
2. Get a PC PSU hotwired to power the riser card through a floppy power connector
3. Get a video card for the PCI-e slot
4. Disassemble the FS6806X, install the riser cable and NVMe flash
5. Add a USB keyboard and monitor

Boot Configuration

It seems that the FX6806X boots from an internal USB flash device containing their ADM installer. We must configure the BIOS to boot from an external USB device for Proxmox installation and from NVMe flash after installation. Asus also recommends disabling their USB flash drive when installing another OS to protect it from accidental deletion, and so that it’s around for recovery later.

1. Plug in Proxmox installer on a USB drive
2. Get into the BIOS by holding F2 during power-on
3. Reorder drives on boot menu so Proxmox installer comes first, then NVMe flash drives, then internal USB flash drive
4. Optional, but recommended: disable internal USB flash drive

• BIOS AMD menu -> AMD CBS -> FCH Common Options -> USB Configuration Options -> USB1 2.0 Port Enable-> Disabled

5. Save and exit BIOS

Add an External 10 GbE NIC and an External Slow NIC

The built-in 10 GbE NIC uses the amd_xgbe driver, which is present in the Proxmox kernel. However, the NIC has to be initialized correctly by the BIOS or a device tree overlay or the driver can’t use it. I also see reports of regressions in driver support for this chipset. After a day of trying to get the built-in 10 GbE NIC to work, I gave up and got an external 10 GbE NIC. Using the external 10GbE NIC requires additional packages, so there’s a catch-22.

Adding an external slow NIC gets around the catch-22 by providing network connectivity to install the necessary packages to support the external 10 GbE NIC. It also provides the comfort of knowing that there’s another way into the box if the 10 GbE NIC has issues later. This alternative was important to me since I wanted to use all six M.2 NVMe bays for storage, and didn’t want the PCIe video card hanging dangerously off to the side.

1. Get a compatible external 10 GbE NIC, such as the IOCREST 10 GbE NIC
2. Get a common, slow USB NIC dongle that Proxmox supports out of the box.

A complication here is that Linux hosts don’t do well with two interfaces on the same LAN. I’m fortunate to have a separate VLAN for PTP where I could address the slow external NIC.

Install Proxmox 9.1

1. Set up the root filesystem and network on the slow USB NIC.
2. Reboot and apt install bolt polkit to get support for Thunderbolt hotplugging devices.
3. boltctl list to get the UUID of the external 10 GbE NIC.
4. boltctl enroll <UUID> to authorize the external 10 GbE NIC.
5. lspci -nn | grep -i ether and expect to see something like

e8:00.2 Ethernet controller [0200]: Advanced Micro Devices, Inc. [AMD] XGMAC 10GbE Controller [1022:1458]

Naming Interfaces

Someday the internal 10 GbE NIC may work, and it has at least two network interfaces until then, so confusion would be likely if naming weren’t clear. I wanted to name the on-board 10 GbE NIC ‘board0’, the external 10 GbE NIC ‘prod0’, and the slow external NIC ‘oob1’ (out-of-band management). Knowing the MAC addresses of each NIC allowed persistent names to be set by creating files in /etc/systemd/network/.

root@pve:~# cat /etc/systemd/network/20-oob1.link
[Match]
MACAddress=3c:18:a0:00:00:01

[Link]
Name=oob1
root@pve:~# cat /etc/systemd/network/30-prod0.link
[Match]
MACAddress=88:c9:b3:00:00:02

[Link]
Name=prod0
root@pve:~# cat /etc/systemd/network/40-board0.link
[Match]
MACAddress=78:72:64:00:00:03

[Link]
Name=board0
root@pve:~# cat /etc/systemd/network/99-disable-predictable-names.link
[Link]
NamePolicy=

Aside from the naming consistency, I wanted the external interfaces to come up at boot, but also if they were hot-plugged later. Udev rules, and the right tweaks to /etc/network/interfaces, make that possible.

root@pve:~# ls -l /etc/udev/rules.d/
total 11
lrwxrwxrwx 1 root root   9 Nov 22 10:29 60-bridge-network-interface.rules -> /dev/null
-rw-r--r-- 1 root root  82 Nov 24 11:34 70-prod0.rules
-rw-r--r-- 1 root root 140 Nov 24 13:32 99-ifupdown-hotplug.rules
-rw-r--r-- 1 root root 116 Nov 24 14:21 99-oob-bridge.rules
-rw-r--r-- 1 root root 168 Nov 24 16:08 99-prod0-bridge.rules
root@pve:~# cat /etc/udev/rules.d/60-bridge-network-interface.rules
root@pve:~# cat /etc/udev/rules.d/70-prod0.rules
ACTION=="add", SUBSYSTEM=="net", ATTR{address}=="88:c9:b3:00:00:02", NAME="prod0"
root@pve:~# cat /etc/udev/rules.d/99-ifupdown-hotplug.rules
ACTION=="add", SUBSYSTEM=="net", NAME=="oob0", RUN+="/sbin/ifup oob0"
ACTION=="add", SUBSYSTEM=="net", NAME=="oob1", RUN+="/sbin/ifup oob1"
root@pve:~# cat /etc/udev/rules.d/99-oob-bridge.rules
ACTION=="add", SUBSYSTEM=="net", ATTR{address}=="3c:18:a0:00:00:01", RUN+="/sbin/ip link set dev oob1 master vmbr2"
root@pve:~# cat /etc/udev/rules.d/99-prod0-bridge.rules
ACTION=="add", SUBSYSTEM=="net", ATTR{address}=="88:c9:b3:00:00:02", \
    RUN+="/sbin/ip link set dev prod0 up", \
    RUN+="/sbin/ip link set dev prod0 master vmbr0"
root@pve:~#

And /etc/network/interfaces:

auto lo
iface lo inet loopback

# Onboard amd_xgbe 10GbE interface that doesn't seem to work with Proxmox 8.4 or 9.1
iface board0 inet manual

# Thunderbolt 10GbE
auto prod0
iface prod0 inet manual
    pre-up ip link set $IFACE up

auto vmbr0
iface vmbr0 inet static
	address 192.168.1.30/24
	gateway 192.168.1.1
	bridge-ports prod0
	bridge-stp off
	bridge-fd 0
	bridge-waitport 15

### Second BACK-DOOR / OOB MGMT
auto oob1
iface oob1 inet manual
    pre-up ip link set $IFACE up

auto vmbr2
iface vmbr2 inet static
    address 192.168.99.33/24
    bridge-ports oob1
    bridge-stp off
    bridge-fd 0

source /etc/network/interfaces.d/*

Cleanup

1. Remove M.2 to PCI-e riser and video card and replace with the last NVMe flash drive.
2. Add rpool mirror to sixth NVMe drive

sgdisk --zap-all /dev/nvme5n1
sgdisk -R=/dev/nvme5n1 /dev/nvme0n1
sgdisk -G /dev/nvme5n1
zpool attach rpool /dev/nvme5n1p3

3. Configure sixth NVMe drive for booting:

proxmox-boot-tool format /dev/nvme5n1p2
proxmox-boot-tool init /dev/nvme5n1p2

Longer Term

I expect that support for the amd_xgbe driver will be added to Proxmox in the future. When that happens, I can remove the external 10GbE NIC, and run Proxmox on the built-in 10 GbE NIC.

Proxmox Mail Notification

Nov 18, 2025 • Proxmox, Checkmk, monitoring, Postfix

Not long after you start monitoring Proxmox with Checkmk, you may see unhandled service problems like this one.

Proxmox has plenty to tell you — a failed backup, a degraded array, a certificate about to expire — but out of the box it has no way to send any of it. The mail piles up in the local Postfix queue and sits there.

What follows is how to give Postfix a route out, using an external SMTP relay.

1. Make sure your Proxmox server has good Internet connectivity and DNS resolution configured.

2. Configure relay services for Postfix to use an external SMTP server to send mail.

• I’ll use Google in this example. You’ll need to set up 2FA with Google and create an App Password.
• Create /etc/postfix/sasl_passwd with content like this:

[smtp.gmail.com]:587    user@gmail.hosted.domain:SixteenCharacterAppPasswordNoSpaces

3. Configure Postfix to use the relay.

• Add lines like these to /etc/postfix/main.cf:

relayhost = [smtp.gmail.com]:587
smtp_use_tls = yes
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
smtp_tls_security_level = encrypt
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt

• Delete a blank line setting relayhost =, if it exists, to suppress a warning.

4. Create Postfix hash db file for sasl_passwd:

postmap /etc/postfix/sasl_passwd
chmod 600 /etc/postfix/sasl_passwd /etc/postfix/sasl_passwd.db

5. Install encryption libraries and certificates necessary for relay communication:

apt-get install libsasl2-modules ca-certificates

6. Restart Postfix to apply changes:

systemctl restart postfix

7. Run the Postfix queue check to verify mail can be sent:

postqueue -f

It might take a few minutes to drain, especially if there are many messages queued.

8. See if there’s anything left in the queue.

postqueue -p

You should see no entries if everything is working.

Monitoring Proxmox with Checkmk RAW

Nov 17, 2025 • Proxmox, Checkmk, monitoring, storage

Proxmox VE and Proxmox Backup Server are both based on Debian Linux, so we can use the Debian-packaged Checkmk RAW Linux agent to monitor them. This procedure also installs the smart plugin to monitor disk health via S.M.A.R.T., which is especially important for Proxmox nodes with NVMe storage.

1. Create a Checkmk host folder to hold Proxmox VE nodes and Proxmox Backup Servers that will be in your cluster.

2. Inside that folder, create host for each Proxmox VE node and Proxmox Backup Server.

Carefully note hostnames as they will be needed later.

3. Download Checkmk agent Debian package to ProxMox node or PBS and install:

• Browse to Setup->Agents->Linux->Packaged Agents.
• Right-click on check-mk-agent_VERSION_all.deb and copy link address.

cd /tmp
wget http://your.checkmk.server/monitoring/check_mk/agents/check-mk-agent_VERSION_all.deb # Paste URL from above here
dpkg -i check-mk-agent_VERSION_all.deb
rm check-mk-agent_VERSION_all.deb

4. Register agent with Checkmk RAW

cmk-agent-ctl register --server your.checkmk.server:8000 --site monitoring --user agent_registration --hostname HOSTNAME_BEING_REGISTERED --password 'AGENT_REGISTRATION_USER_PASSWORD' --trust-cert

Expect to see Registration complete.

5. Get URL of smart plugin for Checkmk agent:

• Browse to Setup->Agents->Linux->Plug-ins.
• Right-click on smart and copy link address.

6. Download smart plugin to Proxmox node:

wget -O /usr/lib/check_mk_agent/plugins/smart http://your.checkmk.server/monitoring/check_mk/agents/plugins/smart # Paste URL from above here
chmod +x /usr/lib/check_mk_agent/plugins/smart

7. Disable smartd because it blocks access by Checkmk:

systemctl disable --now smartd

8. Verify Checkmk agent is working and smart plugin is loaded:

check_mk_agent | grep '<<<smart>>>'

You should see output like:

<<<smart>>>

9. Configure Checkmk RAW to monitor new services from smart plugin:

• Browse to Setup->Hosts->Your Proxmox Node->Host->Run service discovery.
• Review services, and click Accept All.
• Commit and activate Checkmk changes.

Keeping Time with Proxmox and Ntpsec

Nov 16, 2025 • Proxmox, DNS, NTP, monitoring, ntpsec

Tight network time sync requires consistent latency between packet arrival and user space processing. We can’t expect consistent latency inside a container or VM. And even if we could, we wouldn’t want unprivileged containers or VMs to be steering the Proxmox node clock that is shared by all.

My old homelab configuration had many hosts that were configured to expect DNS resolution and NTP from two well-known IP addresses formerly on bare metal hosts that also provided many other services. The other services have since been migrated to a group of high availability LXC containers delivering just one service each. This eliminated cross-service interactions and allowed services to keep running, even in the face of Proxmox node failures.

Making the jump to Proxmox required different treatment for DNS resolution and NTP services. NTP servers can be referenced by name, but DNS resolvers can’t, due to the obvious chicken/egg problem.

The DNS resolver IP address configured into all my homelab hosts could be moved to HA LXCs easily enough. I created PTR records going to dns{1-3} for these addresses to handle any inverse lookups. It seems unlikely that A records would ever be needed, but I added them for symmetry. No changes to /etc/resolv.conf were required, but it was a good opportunity to audit all my host configs.

The privilege constraints above led me to run time sync daemons directly on my Proxmox nodes for consistent latency between network and user space. I also tried running pass-through or “proxy” time sync daemons inside high-availability LXC containers bound to the well-known IP addresses so that I wouldn’t have to reconfigure NTP clients.

That worked, but I didn’t like the jitter added between the bare metal and containerized time daemons. And it turned out to be easy enough to find all clients using my old NTP servers by address: ntpq -c mrulist. That was a small enough list that I just edited all their NTP client configs to specify servers by the names ntp{1-5}, which I added as additional A records for my bare metal Proxmox hosts.

I’ve chosen ntpsec for my sync daemon on the Proxmox cluster nodes. The default systemd-timesyncd.service isn’t really designed to discipline the system clock to match master clocks. Rather, it steps the system clock when the difference to a master clock is above a threshold, while making occasional frequency adjustments below this threshold.

I considered chrony because it runs continuously, measures multiple masters against each other, chooses the best, and disciplines the system clock from it. However, chrony is optimized for TSC discontinuities from CPU speed steps, sleep/wake, and CPU scheduling latency etc. We should not see these discontinuities in Proxmox nodes. Chrony would be great if we couldn’t run a time sync daemon directly on the Proxmox node.

Ntpsec is my preferred time sync daemon because it supports the newer NTS protocol, which authenticates time servers and so prevents man-in-the-middle attacks. Moreover, it directly supports the ntpviz visualization tools that let you easily see how your NTP servers are performing.

apt install ntpsec
apt install ntpsec-ntpviz # Optional

Monitor your NTP server performance using ntpviz: http://ProxmoxHost/ntpviz/