Serving Email Part 1: The Pieces to the Puzzle

by William David Herndon

This was supposed to be a single blog post. But it has expanded until there is no way to do it in a single, reasonable sized post. This post is an introduction to the resources you will need to know for the rest of the series. You can skip the parts of the introduction you're familiar with, without missing anything important.

This series:

The Goal

You have your own website or websites, and you want to provide your users with email. You want your user to enter their email address and their password into their favorite email-client application, and it should just work.

It should also be secure, so that it will continue to "just work" for your customers.

If you are providing the service yourself, you may or may not want to offer your user a choice of account types:

If you're using an ISP that provides email, and it just works, congratulations! You can skip out on this blog post.

Otherwise your goal is that the above should be the end of user setup. Anything else and your users will be calling you for support.

The Problem

So, the user chooses their account type, and instead of it just working, the user is presented with screens like these:

If they get anything wrong, they'll get an error message they can't interpret (and perhaps you can't either).

Or worse. They may get a scary screen like this.

The Pieces of the Puzzle

So, there are a lot of pieces to this puzzle. I have short sections on each linked below.

Which pieces you will want to use depends on your and your user's requirements and limitations. Getting all these pieces to play nice together is a difficult dance, but I'm going to try and walk you through it here.

The Account Type / Email-Protocols

Incoming Protocol: POP - Post Office Protocol

Here the name is the program: the server acts like a post office, and when your client picks up your email, it is deleted from the server. This mode is sometimes called "download-and-delete" or "maildrop". Often called POP3, because version 3 is the last major version, the current formal documentation is RFC 1939).

Historically, this was the only way to do things: you had multiple servers between sender and receiver, and servers were often offline, so each server on the way kept your email until it was picked up by the next station. Now servers usually talk directly to each other.

This may still be a good protocol choice if you want to minimize the amount of space email takes up on the server, or if your users don't want copies of their email on the server. On the other hand, if your users want to access their email from their mobile, their tablet and their desktop without missing a beat, then they'll need the emails to stay on your server where they can access them. That's where the major alternative, IMAP, comes in.

A modern POP3 server listens on port 110 and may thence switch to encrypted protocol, or it may listen for encrypted traffic directly (POP3S) on port 995, or both.

Incoming Protocol: IMAP - Internet Message Access Protocol

This protocol allows a client to manipulate messages (move between folders, copy, delete) and retrieve messages stored on a server. Messages are only deleted on the server when explicitly commanded to do so, and the client normally only keeps a small cache of messages. The current version of this protocol, version 4, was formalized in 2003 with RFC 3501.

Depending on space requirements, you may need to agree with your users on an archiving scheme, so that emails that they need access to remain online, while older emails are stored in an offline archive and deleted from the server.

A modern IMAP server listens on port 143 and may thence switch to encrypted protocol, or it may listen for encrypted traffic directly (IMAPS) on port 993, or both.

Outgoing Protocol: SMTP - Simple Mail Transfer Protocol

This protocol is for sending messages and is used by all major mail servers. The current version was formally defined in RFC 5321.

Although SMTP servers listen on port 25, this is usually reserved for unencrypted traffic between servers. They listen for user email-clients using encrypted traffic directly (SMTPS) on port 587 or port 465 (deprecated), or both.

Microsoft Exchange Server / WebDAV - Web Distributed Authoring and Versioning

This proprietary system from Microsoft works like the IMAP/SMTP combination: messages are kept on the server and it allows you to send messages. WebDAV is a generic method of manipulating data via HTTP, that Microsoft extended for manipulating mail messages on Exchange Server. WebDAV support was discontinued in Exchange Server 2010, replaced by Exchange Web Services (EWS), a SOAP/XML based API.

Since Microsoft Exchange Server supports POP3, IMAP and SMTP anyway, I'm going to leave out this proprietary nightmare that would expand this already too large article.

The Email Server

The big four Mail Server Softwares, often called Message Transfer Agents (MTA), are Exim (ca. 50% market share), Postfix (ca. 30%), sendmail (ca. 8%), and Microsoft Exchange Server (ca. 3%). All except the Exchange Server are Open Source. Depending on your setup, you may not have any choice in which MTA you use. Those that have a choice are usually using a virtual private server (VPS), or a dedicated server.

If you have a choice, any of the four will do - they all support POP3, IMAP and SMTP.

sendmail

The oldest of the three, this Open Source software was first released in 1983. It has great flexibility, but configuring it is daunting.

Postfix

Originally released in 1998 as a replacement for sendmail, Postfix is still under active development. It has integrated protection against spambots and malware. It is released under the IBM Public License.

Exim

The default email program on Debian/Linux distributions, it has over 50% market share. It is often used together with cPanel, a major web-hosting control panel.

Microsoft Exchange Server

Microsoft Exchange Server is proprietary software. In addition to the standard protocols, it supports its own protocol, as mentioned above. It runs only on Windows Operating System, and supports a calendar and other functions as well as email.

The Email Clients

The Big Clients

You probably want to support these:

  • Apple iPhone/iPad - the largest by market share, no choice for those that own one
  • GMail - not just for Google Email accounts
  • Microsoft Outlook - still many people using Windows 7/8 desktop
  • Mozilla Thunderbird - for Linux users and picky Windows users
  • Windows 10 Mail - comes with Windows 10

WebMail

You probably also want to let your users access email via browser with a WebMail program: e.g. Horde, SquirrelMail, RoundCube.

Smaller Clients

You may or may not want to support these smaller clients (I use K-9 Mail myself):

  • Android clients: K-9 Mail, MailDroid, Blue Mail, et al.
  • Desktop clients: Claws Mail, TheBat, et al.
No Worries

There are some popular email clients that have restrictions preventing their use with the mail servers we're talking about here. That is not always obvious: e.g. I installed Touchdown to test it, before discovering I couldn't use it. You won't need to worry about:

  • Touchdown - only supports Microsoft Exchange Server
  • Boomerang - only GMail accounts
  • Yahoo! Mail - only Yahoo accounts
  • AOL Mail - only AOL accounts

DNS - Domain Name System

The Domain Name System has different types of records. In rough order of frequency of usage:

  • A - maps names to IP-Adresses
  • NS - for authoritative name servers
  • CNAME - for aliases
  • TXT - miscellaneous text records:
    • Verification records: e.g. Google, Office 365
    • SPF - Sender Policy Framework - should be TXT duplicate of SPF record.
  • MX - the "Mail Transfer" record, giving the hostname of the mail server
  • SRV - The service record, telling the hostname and port number of specified services
  • SPF - Sender Policy Framework gives info about your email policy to others. This may be a duplicate of an SPF record.

Linux has several commands for looking up DNS records. The most common is host. Under Windows you use NSLookup.

In either case, you should use a lookup program to verify that your record changes are correct and are being propagated. Best is if you can check the records on your server itself first: propagation to other servers may take hours or even days.

Autoconfiguration

Several Email client writers have tried to mitigate the difficulties of configuring their clients by documenting methods of specifying to their client how they should be configured.

Outlook Autoconfiguration

One can tell Outlook how to configure emails by placing an XML-file with the necessary information in a specific location on the main domain, or on an "autodiscover" subdomain. One can also place a redirect on the "autodiscover" subdomain to a secure server or make a SRV-record in DNS. The XML file can be static, or it can be generated dynamically at each request - the file request is done via POST with the username passed. The details are here.

Mozilla Thunderbird Autoconfiguration

Similar to Outlook, one can place an XML file in a specific location, but the location is different: either on the server specified in the DNS MX-record or in the "autoconfig" subdomain. Only a static file is expected, but it supports some variable names, like "%EMAILADDRESS%". The details are here.

SSL/TLS and Certificates

The original email did not have any security, but now it is an absolute requirement: with so many spammers, email servers must properly authenticate everything over a secure connection.

Depending on how you configure your server(s), you may have many subdomains, e.g.: pop3.mydomain.com, imap.mydomain.com, smtp.mydomain.com, webmail.mydomain.com, autodiscover.mydomain.com, autoconfig.mydomain.com. If so, you will either need a wildcard certificate or separate certificates for each subdomain.

Copyright © 2016 William David Herndon. All rights reserved.

This series: