Running a Mail Server on a VPS: What It Really Takes

You install Postfix on a fresh VPS in ten minutes, send a test message to Gmail, and it never arrives. The mail log shows one of two lines: “Connection timed out” on port 25, meaning your provider blocks outbound mail, or a rejection with code 550-5.7.25 because your IP address has no reverse record. That’s the first truth about running your own mail server: the software is the easy part. The hard part is convincing receiving servers that you’re a trustworthy sender, then staying that way every single day. This guide covers what that really takes, in the order you’ll need it.

01Before you install anything: three things your provider controls

Three things can’t be fixed by any setting on your server, because they belong to whoever owns the IP address. Check them on day one, before you invest a single hour in configuration:

• Outbound port 25: Many VPS providers block it by default on new accounts to curb spam. Some open it on request once they’ve verified who you are; some never open it. Ask before you buy: without it, your server can’t deliver a single message to another server.
• Reverse DNS: A PTR record maps an IP address back to a name, and it’s set by whoever owns the address block, meaning your provider, from their control panel or through a support ticket. It must return your mail server’s own name, not a generic one the provider generated.
• IP reputation: The address you were given may have sent spam before you got it and landed on blocklists such as Spamhaus. A clean new address has no reputation at all, so receiving servers treat it with caution for its first few weeks.

# Does port 25 get out? “succeeded” means yes; a timeout means it’s blockednc -vz -w 5 gmail-smtp-in.l.google.com 25# The PTR record for your server’s IP: it must return the mail server’s namedig -x 203.0.113.25 +short# The forward direction: the name must return the same addressdig +short A mail.barq-bahrain.com# Is the address listed by Spamhaus? No output means not listed (octets reversed)dig +short 25.113.0.203.zen.spamhaus.org

02The software stack and server size

A complete mail server has four roles, and each has a program proven in production:

• Postfix: The mail transfer agent: it receives mail on port 25, accepts your users’ outgoing mail on submission ports 587 and 465, and delivers it to servers around the world.
• Dovecot: Stores the mailboxes and serves them to users’ mail apps over IMAP on port 993, and checks passwords on Postfix’s behalf.
• Rspamd: Scans incoming mail and classifies spam, signs outgoing mail with your domain’s key, and keeps its statistics in Redis.
• ClamAV: Optional: scans attachments for malware, and on its own takes about 1 to 1.5 GB of memory.

# Postfix asks for a configuration type: choose Internet Site and enter your domainsudo apt updatesudo apt install -y postfix dovecot-imapd dovecot-lmtpd rspamd redis-server certbot# Pass mail through Rspamd (it listens on 11332 by default)sudo postconf -e "smtpd_milters = inet:127.0.0.1:11332"sudo postconf -e "non_smtpd_milters = inet:127.0.0.1:11332"sudo postconf -e "milter_default_action = accept"sudo systemctl reload postfix# Open only the mail ports, plus 80 if certbot issues the certificate on this serversudo ufw allow 25,465,587,993/tcpsudo ufw allow 80/tcp

Next, enable the submission and submissions services in master.cf, require authentication and encryption on both, and connect Postfix to Dovecot for authentication and LMTP delivery. If you’d rather not wire the pieces together by hand, all-in-one bundles such as Mailcow on Docker, Mail-in-a-Box or iRedMail install them together with sensible defaults. They save you days of setup, but they don’t exempt you from port 25, PTR and reputation, or from updates and monitoring.

Disk size is set by the mailboxes, not the software: 25 mailboxes with a 5 GB quota means 125 GB, so add a margin of about 20% for indexes, logs and growth, and expand the NVMe storage as you approach it.

03Server identity: one name in three places

Receiving servers check that your server introduces itself with a stable, consistent name. Pick a name such as mail.barq-bahrain.com and make it appear, identically, in three places: an A record pointing from the name to the address, a PTR record pointing the address back to the name, and the HELO greeting Postfix opens every connection with. This match is called forward-confirmed reverse DNS, and its absence alone is enough for Gmail to reject your mail.

Watch out for IPv6. If your server has an IPv6 address, Postfix will prefer it with servers that support it, such as Gmail, and Gmail requires a PTR for every address that sends to it. Either set a PTR for the IPv6 address and add it to SPF, or restrict Postfix to IPv4 and don’t publish an AAAA record for the mail host until you’re ready. Otherwise you’ll see intermittent rejections that look random: what leaves over IPv4 gets through, what leaves over IPv6 bounces.

# The name Postfix announces in HELO; it must match the PTRsudo postconf -e "myhostname = mail.barq-bahrain.com"# Send over IPv4 only until the IPv6 address has a PTRsudo postconf -e "inet_protocols = ipv4"sudo systemctl restart postfix

04Authentication: SPF, DKIM and DMARC

These three records answer one question for receiving servers: did this message really come from the domain’s owner? SPF lists the addresses allowed to send, DKIM signs every message with a private key on your server, and DMARC requires one of them to align with the domain shown in the From header, and says what to do with the message if neither does. Here are the records for a domain with a single mail server:

End SPF with ~all while you test, then -all once you’re sure your server is the only sender. And never publish more than one SPF record for a domain: two records make the whole check fail. Rspamd generates the DKIM key and prints its record ready to publish:

# A 2048-bit RSA key, with a selector that carries the year so it’s easy to rotate latersudo mkdir -p /var/lib/rspamd/dkimsudo rspamadm dkim_keygen -d barq-bahrain.com -s s2026 -b 2048 -k /var/lib/rspamd/dkim/barq-bahrain.com.s2026.keysudo chown -R _rspamd:_rspamd /var/lib/rspamd/dkim# Tell Rspamd where the key is and which selector to use, then reload itsudo tee /etc/rspamd/local.d/dkim_signing.conf >/dev/null <<"EOF"path = "/var/lib/rspamd/dkim/$domain.$selector.key";selector = "s2026";EOFsudo systemctl reload rspamd

Start DMARC at p=none with an rua address, and the large mailbox providers will send you daily aggregate reports. Read them for two or three weeks to learn everything that sends as your domain, such as your website’s forms or your invoicing system, then move to p=quarantine and finally p=reject. Only at that last stage does spoofing your domain actually get rejected.

What Gmail and Yahoo have required since 2024

• Every sender: Authentication with SPF or DKIM, a valid PTR record, and a spam-complaint rate under 0.3%. Gmail also requires mail to reach it over TLS.
• Senders of about 5,000 messages a day or more: Both SPF and DKIM, a DMARC record (p=none is enough), a From domain aligned with one of them, and one-click unsubscribe in marketing mail.
• Microsoft: Applied similar rules in 2025 to high-volume senders to Outlook.com addresses.

05Encryption in transit: TLS and MTA-STS

Between mail servers, the connection starts in plain text on port 25, and the STARTTLS command upgrades it to an encrypted one if both sides support it. That’s opportunistic encryption: if an attacker on the path strips the STARTTLS offer, the sending server carries on unencrypted. So you need two things: a valid certificate for your server’s name, and an MTA-STS policy telling senders to deliver your mail only over TLS with a valid certificate.

Get the certificate from Let’s Encrypt with certbot and automate its renewal, because public certificate lifetimes keep shrinking: 200 days at most since March 2026, and 100 days from March 2027. In Postfix and Dovecot, make TLS 1.2 the minimum. The stronger alternative to MTA-STS is DANE with TLSA records, but it requires a DNSSEC-signed zone.

# 1) A TXT record announcing the policy; change id whenever you edit it_mta-sts.barq-bahrain.com.  TXT  "v=STSv1; id=20261005"# 2) The policy file, served over HTTPS with a valid certificate at:# https://mta-sts.barq-bahrain.com/.well-known/mta-sts.txtversion: STSv1mode: testingmx: mail.barq-bahrain.commax_age: 604800# 3) Daily reports of TLS failures_smtp._tls.barq-bahrain.com.  TXT  "v=TLSRPTv1; rua=mailto:[email protected]"

06Testing, monitoring and backups

Before you hand the addresses to your team, test from a machine outside the server:

# The domain’s records as the world sees themdig +short MX barq-bahrain.comdig +short TXT barq-bahrain.comdig +short TXT s2026._domainkey.barq-bahrain.comdig +short TXT _dmarc.barq-bahrain.com# TLS on the submission port: protocol version and certificateopenssl s_client -starttls smtp -connect mail.barq-bahrain.com:587 -servername mail.barq-bahrain.com -brief </dev/null# A test message with authentication and STARTTLS; swaks prompts for the passwordswaks --to [email protected] --from [email protected] --server mail.barq-bahrain.com --port 587 --tls --auth LOGIN --auth-user [email protected]

Open the message in Gmail and choose “Show original”: you should see PASS next to SPF, DKIM and DMARC. Then watch the server daily during its first month, and weekly after that:

• Mail queue: A growing queue means receiving servers are deferring or rejecting your mail. Alert yourself if it passes a few dozen messages.
• Reputation: Read your DMARC reports, register your domain in Google Postmaster Tools to see your complaint rate and IP reputation, and check blocklists every week.
• Updates: Turn on automatic security updates: a mail server exposed to the internet is a permanent target for password-guessing attempts.

# Queue size: the last line gives the number of waiting messagespostqueue -p | tail -n 1# Deferred and bounced messages in the last hour, with their reasonsjournalctl -t postfix/smtp --since "1 hour ago" | grep -E "status=(deferred|bounced)"# Retry delivery once you’ve fixed the causesudo postqueue -f

Backups here go beyond the message folder: the mailboxes, the Postfix, Dovecot and Rspamd configuration, and the DKIM keys. A lost key can be regenerated and its record republished; lost messages are gone for good. Copy them every night to somewhere off the server, and test a restore once a month.

ProblemMail doesn’t leave the server, and the log says Connection timed out on port 25

FixYour provider blocks outbound port 25. Ask them to open it, or send through a trusted SMTP relay on port 587 if the block is permanent.

ProblemGmail answers with code 550-5.7.25

FixThe address has no PTR, or the PTR doesn’t match the A record. Ask your provider to set it to your server’s name, and check the IPv6 address too.

ProblemGmail answers with 550-5.7.26 because the sender is unauthenticated

FixSPF doesn’t cover your address, or no DKIM signature is being added. Check both records with dig, and make sure outgoing mail passes through Rspamd.

ProblemAuthentication is all PASS, but messages land in the spam folder

FixThe problem is reputation, not configuration. Start with small volumes to correspondents who reply, and watch Postmaster Tools and the blocklists.

07When is a managed service the better choice?

A self-hosted server makes sense in specific cases: automated mail from your applications at volumes you know, requirements that keep messages on a server you run, or a technical team with time to monitor it. For staff mailboxes your business depends on, count the real cost: two to four hours a week of monitoring and updates, weeks to rebuild reputation if your address lands on a blocklist, and a queue piling up over a holiday weekend with nobody watching.

If you decide to run your own server, ask about port 25 and PTR before you buy, publish SPF, DKIM and DMARC before the first message, and watch the queue and the reports daily during the first month. If no one on your team can take that on, business email from Illevante in partnership with Acrux Mail, coming soon, gives you addresses on your domain without carrying an IP address’s reputation on your shoulders. You can also combine the two: team mailboxes on a managed service, and your applications’ automated mail from your own server.