DRBD with MySQL CentOS 8: A Practical, No‑Nonsense Guide to Data Mirroring
MySQL on a single node is fine, but if a server crashes you’re left scrambling for backups or pulling the database back online from scratch. In this post I’ll show you how to pair DRBD (Distributed Replicated Block Device) with MySQL on CentOS 8 so that writes go to two disks at once and your data stays safe even when one machine goes down.
1. Why DRBD? What It Does for You
DRBD replicates block devices over the network, acting like a “live copy” of your entire disk partition. For MySQL it means every transaction lands on both nodes almost instantly—no need to configure replication at the SQL level or worry about binary logs falling behind.
I’ve seen this happen after a bad driver update: one node’s kernel panicked during an upgrade, wiping out its MySQL data directory, while the other still had a pristine copy. DRBD saved me a week of re‑importing.
If you’re only worried about simple file backups or don’t mind a small lag, plain rsync is cheaper. But for a production database that can’t afford downtime, DRBD gives you real redundancy.
2. Pre‑flight Checklist
1. Two CentOS 8 servers with at least one NIC each (one used for DRBD traffic).
2. MySQL 8.x installed on both nodes (the same version is required).
3. Root or sudo access everywhere.
4. A stable, low‑latency network link—DRBD copies every byte as it arrives.
3. Install DRBD and Required Packages
sudo dnf install -y drbd90-utils device-mapper-persistent-data lvm2
We’re using the drbd90 package because it ships with newer kernel support. The LVM tools are handy later when we create thin pools for the replicated data.
4. Create a DRBD Resource Configuration
On both servers, make /etc/drbd.d/mysql.res. Replace 10.0.0.1 and 10.0.0.2 with your actual IPs:
resource mysql {
protocol C; # synchronous replication (C for "concurrent")
on node1 {
device /dev/drbd0;
disk /dev/mapper/vg_mysql-lv_mysql;
address 10.0.0.1:7789;
meta-disk internal;
}
on node2 {
device /dev/drbd0;
disk /dev/mapper/vg_mysql-lv_mysql;
address 10.0.0.2:7789;
meta-disk internal;
}
}
> Why the C protocol? Because with MySQL you want writes to be committed on both nodes before reporting success; a faster “D” mode would risk data loss if one node dies mid‑write.
5. Create LVM Volumes for DRBD
We’ll create a thin pool so that if the disk shrinks, we can resize easily.
sudo pvcreate /dev/sdb # assume /dev/sdb is spare on both nodes sudo vgcreate vg_mysql /dev/sdb sudo lvcreate --thinpool mysql-thin -l 100%FREE vg_mysql
Now set up the thin volume that DRBD will use:
sudo lvcreate --thin lv_mysql --virtualsize 200G -T vg_mysql/mysql-thin
This gives us a 200 GB logical volume that DRBD can mirror.
6. Initialize DRBD
On node1 (the initial primary):
sudo drbdadm create-md mysql # build meta data once per node sudo drbdadm up mysql # bring the resource online
On node2, just drbdadm up mysql. Now both nodes should report a state of “Primary/Secondary” for /dev/drbd0.
7. Format and Mount the Replicated Device
sudo mkfs.xfs /dev/drbd0 sudo mkdir -p /var/lib/mysql sudo mount /dev/drbd0 /var/lib/mysql
Add it to /etc/fstab so that it mounts on boot:
/dev/drbd0 /var/lib/mysql xfs defaults,_netdev 0 0
> The _netdev option tells the system to wait for network services before mounting, which is essential because DRBD relies on networking.
8. Install MySQL and Point It at the New Data Directory
If you haven’t installed MySQL yet:
sudo dnf install -y @mysql
Edit /etc/my.cnf.d/server.cnf (or mysqld.conf) and set:
[mysqld] datadir = /var/lib/mysql innodb_file_per_table = 1
Stop MySQL, copy any existing data over, then start it again:
sudo systemctl stop mysqld sudo rsync -avz /path/to/old_mysql_data/ /var/lib/mysql/ sudo systemctl start mysqld
Verify that mysql is running and can read/write by connecting locally.
9. Verify DRBD Sync Status
Run on both nodes:
drbdadm status mysql cat /proc/drbd
You should see the state as “Primary/Secondary” (or “UpToDate”) and the sync percent at 100%.
If you trigger a write, the other node’s log will show a write event. This is how you know replication is happening in real time.
10. Handling Failover
When one server goes offline:
1. On the surviving node, run:
sudo drbdadm primary --force mysql
2. If MySQL was running on the failed node, it will stop automatically because its data store disappeared.
3. The surviving node can now serve read/write traffic.
When the downed server comes back:
sudo systemctl start mysqld # if you’re restoring the old data sudo drbdadm secondary mysql # make sure it stays a replica, not primary
You’ll see the sync progress as it catches up. Because we used the C protocol, the node that comes back will never be in a state where its data diverges from the live copy.
11. Common Pitfalls to Avoid
- Wrong protocol: Switching to D after you’ve already set up may cause silent data loss during an outage. Keep it at C.
- Disk size mismatch: The thin volume must be big enough for MySQL’s growth; otherwise, the replica will fail mid‑sync. Resize with LVM if needed.
- Network hiccups: DRBD treats any packet loss as a serious error. If you’re on an unreliable LAN, consider adding net.core.rmem_max=1048576 to both nodes and tuning the rdisk options.
12. Tuning for Performance (Optional)
If your workload is read‑heavy and writes are relatively light, you might switch to a “read‑only” secondary:
drbdadm adjust --use-raw-clients mysql
But be careful: read‑only nodes can’t accept write traffic; they’re purely for backup or failover.
13. Final Thought
DRBD turns two ordinary CentOS 8 servers into a single, fault‑tolerant MySQL appliance. The initial setup is a bit involved, but once it’s up and running you’ll have the peace of mind that your data never really goes offline, and you won’t need to wrestle with replication drift or binary logs.
Give it a shot on a spare pair of boxes; if you run into hiccups, drop me a line. Happy mirroring, and keep your databases humming while your coffee stays warm!
