Skip to content

VPS Storage: From NVMe to Backups That Restore

Section 1 of 7
Size 3 of 5

Virtual servers04/08

VPS Storage: From NVMe to Backups That Restore

What RAID 10 protects you from and what it doesn’t, how to monitor and grow a disk, apply the 3-2-1 rule with restic, and test restores every month.

Abdulwahed MohammadContent writer at tech+9 min read

Your server’s disk fills up at 3 am because application logs grew by 20 GB in a week; the database stops writing and the store goes down. Or worse: a wrong command deletes the images folder, and you discover that copied the delete to all four disks at the same instant. VPS storage isn’t managed by buying more space. It’s managed by answering three questions: what sits under your disk, when will it fill up, and how do you get your data back when it’s gone. We answer them here, with commands.

01What sits under your virtual disk

Your server sees a virtual disk, usually named /dev/vda, presented by the hypervisor through virtio. Underneath is an array of physical disks run by the provider, which you neither see nor manage. But its type sets what you can expect from it:

  • NVMe vs SATA SSD: NVMe drives connect to the CPU over the PCIe bus and handle requests in many parallel queues, so latency stays low while the database reads and writes at the same time. SATA goes through an interface designed for spinning disks, with a single queue of 32 commands.
  • RAID 10: Data is mirrored across pairs of disks, then striped across the pairs. The array survives the loss of one disk in each pair, and writes stay fast because there’s no parity to compute, unlike RAID 5 and RAID 6.
  • A per-server cap: Providers usually cap read and write operations per second for each server, so one server can’t monopolise the array. We explain how to measure it in a separate article on VPS vs VDS.

CoreVPS servers, for example, run on NVMe storage in software RAID 10. That protects you from a failed physical disk, and from nothing else, as you’ll see. Start by finding out what your system sees:

The storage map as your server sees it
# Disks, partitions, their filesystems and mount pointslsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS# The root filesystem and its mount optionsfindmnt /

02Filesystems and LVM

On an NVMe disk you won’t feel much performance difference between ext4 and XFS for an ordinary site or store. The real difference is in management:

Criterionext4XFS
Default onUbuntu and DebianAlmaLinux and Rocky Linux
GrowingWhile mountedWhile mounted
ShrinkingPossible after unmountingNot possible
StrengthMature and simple, with well-known repair toolsLarge files and parallel writes
InodesCount fixed at creationAllocated as needed

The practical rule: stay with what your system ships. The exception is a server that stores very large files or writes heavily in parallel, such as a video archive, which may benefit from XFS.

is a layer between the disk and the filesystem that pools disks into one group and carves it into logical volumes you can grow without repartitioning. On a single-disk server you usually don’t need it, but it helps when you add a second data disk, or want the database directory in its own volume that logs can’t fill.

Most important in practice: when you upgrade disk space from the control panel, the virtual disk grows, but the partition and filesystem keep their old size until you extend them. Some templates do this automatically at boot through cloud-init; if yours didn’t, here are the commands:

Growing the partition and filesystem after a disk upgrade. Take a backup first, and check the partition name before every command
# Check the new disk size and the root partition number (vda1 here)lsblk# Grow the partition to fill the disk (the tool comes in the cloud-guest-utils package)sudo growpart /dev/vda 1# Then grow the mounted filesystem: for ext4sudo resize2fs /dev/vda1# Or for XFS, by mount point rather than partition namesudo xfs_growfs /# If the root is on LVM: resize the physical volume, then the logical volume with its filesystem (check names with lvs)sudo pvresize /dev/vda3sudo lvextend -r -l +100%FREE /dev/ubuntu-vg/ubuntu-lv

03Watch it before it fills

A disk doesn’t fill up suddenly: it fills quietly, then everything stops at once. Three sources fill server disks more than anything else: logs, forgotten local backups, and millions of small files such as sessions and stale cache, which use up before they use up space. These commands tell you where the space goes:

Where the space goes: filesystems, directories, open files, logs and disk utilisation
# Space per filesystem, then inodesdf -hTdf -i# The largest directories under /var, without crossing into other filesystemssudo du -xh --max-depth=2 /var | sort -h | tail -15# An interactive space explorer (install the ncdu package)sudo ncdu -x /# Deleted files still held open by a process, so their space isn’t freedsudo lsof +L1# The systemd journal’s size, then trim it to 500 MBjournalctl --disk-usagesudo journalctl --vacuum-size=500M# Disk latency and utilisation, every two seconds, five timesiostat -x 2 5

Trimming the journal once isn’t enough: set a permanent limit with the SystemMaxUse key in journald’s settings, and make sure logrotate rotates your applications’ logs. Then set an alert at 80% of space and 80% of inodes, from your server panel or a monitoring tool such as Prometheus or Zabbix; an alert at 95% comes too late.

A phone on a bedside table in a dark bedroom at night, its screen lit by a notification
An alert at 80% of space and inodes gives you time to act before the disk fills up at 3 am.
Problemdf says the disk is full, but du finds only half the space
FixA deleted file is still held open by a process, such as a log you removed by hand while the web server keeps writing to it. List it with lsof, then restart the service to free the space.
Problem“No space left on device” with 30% free space
FixYou’ve run out of inodes, not space. Check the IUse% column in df’s output, and look for a directory with millions of small files, such as PHP sessions or a stale cache.
ProblemThe disk fills up every night and empties by morning
FixThe backup script writes its dump locally before uploading it. Compress the dump, stream it straight out, or give it enough space.

04A snapshot is not a backup

A is an image of the disk’s state at a moment in time, taken in seconds because it doesn’t copy the data but records what changes afterwards. It’s excellent before an OS upgrade or a risky update, because rolling back is fast. But it lives on the same storage system, so if that system is lost, the snapshot goes with it.

A storage server’s front in a data-centre rack: two rows of drive bays with blue and green activity lights
A snapshot lives on the same storage system that holds your disk, so if that system is lost, the snapshot goes with it.

And mind the database: a disk snapshot taken while it’s writing is like pulling the power cord. InnoDB usually recovers, but it’s safer to take a consistent dump alongside it with mysqldump or a tool such as mariadb-backup. Here’s what each option protects you from:

RiskRAID 10SnapshotBackup outside the server
Physical disk failureProtectsSurvives if the array doesProtects
Accidental delete or broken updateDoesn’t protectProtects if taken before itProtects
Ransomware encrypting your filesDoesn’t protectProtects if the attacker can’t reach itProtects if the server can’t delete it
Loss of the server or the storage systemDoesn’t protectDoesn’t protectProtects
Compromise of your provider accountDoesn’t protectDoesn’t protectProtects if it’s with another party

Every CoreVPS plan includes backup space outside the server, from 10 to 100 GB depending on the plan, where you save a copy of your server in one click. That’s your second copy, on a system other than your server’s disk. The third must live with another party altogether.

05The 3-2-1 rule in commands

The is simple: three copies of your data, on two different kinds of storage or two independent systems, one of them in another location. Modern practice adds two more numbers: one copy that can’t be modified or is offline, and zero errors in restore testing. On a photo-archive server in Hawalli, arshif-hawalli.com, it looks like this:

  • Copy one: The live data on the server.
  • Copy two: A daily copy of the server in the provider’s backup space.
  • Copy three: An encrypted restic repository on a server in another data centre or at another provider, daily, with weekly and monthly retention.

restic splits files into chunks, uploads only what changed, and encrypts the repository by default. Keep its password outside the server too: lose it and you lose every backup.

A daily encrypted restic backup to another server, with a consistent database dump
sudo apt install -y restic zstd# The rest runs as root, e.g. from a scheduled job: the repository is on another server over SFTPexport RESTIC_REPOSITORY=sftp:[email protected]:/srv/restic/arshif-hawalliexport RESTIC_PASSWORD_FILE=/root/.restic-pass# Once only: create the repositoryrestic init# A consistent database dump first, then back up files, settings and the dumpmysqldump --single-transaction --quick --all-databases | zstd > /var/backups/db.sql.zstrestic backup /var/www /etc /var/backups# Keep seven daily, four weekly and six monthly snapshots, and remove the restrestic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
A mirror with rsync: useful for moving files or keeping a standby server ready, but not an archive
# A quick mirror of the uploads folder to another server. Careful: --delete propagates deletions toorsync -aH --delete /var/www/arshif-hawalli/uploads/ [email protected]:/srv/mirror/uploads/

rsync with deletion turned on is a mirror, not an archive: delete a file by mistake and the mirror deletes it on the next run. Use it for transfers or a warm standby, and rely on restic to keep history.

And don’t let the production server hold the right to delete its own backups: an attacker who gets in will look for them first. Have the backup server pull the copies itself, or run the restic repository through rest-server in append-only mode, or turn on Object Lock if you use S3-compatible storage. Then run pruning and deletion from the backup server itself.

Inside a backup tape library in a data centre: rows of tape cartridges and a robotic arm moving between them
The copy the production server can’t delete, on a server that pulls it or on offline media like these tapes, is the one that survives a break-in.

06Test the restore before you need it

A backup that has never been restored is a hope, not a backup. Once a month, set aside half an hour for this drill on a test server or a temporary folder, never on top of production:

The monthly restore drill: check the repository, restore the files, import the database on a test server
# Check the repository’s integrity, actually reading 5% of its datarestic check --read-data-subset=5%# The snapshots available and their datesrestic snapshots# Restore the latest snapshot into a temporary folder, not over productionrestic restore latest --target /tmp/restore-testls -la /tmp/restore-test/var/www# On a test server only: decompress and import the dump, then check an important tablezstd -dc /tmp/restore-test/var/backups/db.sql.zst | mysql

And note how long the full restore took: that number is the shortest outage you’ll live through on the day of a real failure. If it’s longer than your business can tolerate, the fix lies in the architecture, not the backup tool.

07Practical recommendation

For a typical VPS this is enough: stay with the default filesystem, alert at 80% of space and inodes, and cap the systemd journal. Take a snapshot or a provider-panel backup before every major update, run restic daily to another location with permissions that don’t let the server delete its backups, and test a restore every month.

All of it is an hour of work on day one, plus a half-hour drill each month. It’s the difference between an outage that costs you an hour and one that costs you the project.

Test your understanding

Three questions, one minute

⁦0 / 3⁩Set 1

  1. 01Someone accidentally deleted the images folder on a server with NVMe storage in RAID 10. What brings it back?

  2. 02df says the disk is full, but du finds only half the space. What’s the likeliest explanation?

  3. 03“No space left on device” appears with 30% free space. What do you check?

Ready to start? Power your projects today

Choose the hosting, server or domain that suits your business, and our specialist team is available around the clock to make your projects a success.

A .com domain a year
$17
cPanel hosting per month
$2
CoreVPS servers per month
$3.50