Serving Email Part 6: Debugging

This is the final post in the now six part series "Serving Email". In this post we will explore how to figure out what's wrong when your email server isn't working properly.

This series:

Part 1: The Pieces of the Puzzle
Part 2: Subdomains and DNS
Part 3: Encryption and SSL Certificates
Part 4: The Email Server(s)
Part 5: Autoconfiguration
Part 6: Debugging Email Problems <- You are here

sign post pointing in 3 different directions

Where to Start

There are several parts to test when testing an email server. At the most basic level:

Can you send emails from the server?
Can you receive emails sent to the server?

Each of these can be broken down further:

Can the service be located: is the meta information (DNS, et al.) about the service correct?
Can the appropriate service be connected to on the appropriate socket?
Is the SSL-certificate correct? (correct name? correct dates? etc.)?
Does a secure connection occur?
Is the authentication information correct?
Does autoconfiguration set things correctly?

Also: is the MTA (SMTP/outgoing server) storing the emails in the same format and location that the IMAP/POP3/incoming server is expecting?

This is my process (YMMV):

Set up Thunderbird as a client to the mail server
- if it autoconfigures, I verify the parameters
- if needed correct the parameters manually
Send myself an email (server -> itself)
Once that works, I try sending an email from another server
Once that works, I try sending an email to another server
Verify that autoconfiguration works for Thunderbird, Outlook
Test Configuration Profile for iPhone

I leave fixing autoconfiguration for last, even though it occurs first: I can always configure manually until it works. At any other point where it fails, I check likely failure points.

If the connection fails it is usually obvious in the test phase of your email client configuration. I try manually connecting using the appropriate tool (see section below) and if that succeeds, manually authenticating. Possible causes are misconfiguration of the server and firewall interference.

More commonly (in my experience) the connection works, but it fails to send or receive mails. The most common cause for that is misconfigured DNS entries.

Testing Connections

If you're having trouble receiving emails or connecting as a client, you'll want to test if the appropriate port for the protocol is open and you can connect correctly.

I usually test the unencrypted connection before testing the encrypted connection - you can go straight to the encrypted connection (below), but if the server supports the unencrypted ports (STARTTLS starts out unencrypted) I like to use telnet to make sure that the ports are open and the server responding.

There are several programs that you can use to test the connections to a selected port that you need for testing.

telnet - Unencrypted Only

telnet is an ancient program (I used it in college over 30 years ago) that only gives you a simple unencrypted TCP/IP connection. It is available under both Linux and Windows by default.

Example testing the connection to SMTP (port 25):$ telnet mail.example.com 25 220 mail.example.com ESMTP Postfix (Ubuntu) EHLO example.com 250-example.com 250-PIPELINING 250-SIZE 10240000 250-ETRN 250-STARTTLS 250-AUTH DIGEST-MD5 CRAM-MD5 PLAIN LOGIN 250-ENHANCEDSTATUSCODES 250-8BITMIME 250 DSN QUIT 221 2.0.0 Bye Connection to host lost.
Example testing the connection to IMAP (port 143):$ telnet example.com 143 * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA AUTH=CRAM-MD5 AUTH=CRAM-SHA1 AUTH=CRAM-SHA256 AUTH=PLAIN IDLE ACL ACL2=UNION STARTTLS] Courier-IMAP ready. Copyright 1998-2015 Double Precision, Inc. See COPYING for distribution information. CAPABILITY * CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA AUTH=CRAM-MD5 AUTH=CRAM-SHA1 AUTH=CRAM-SHA256 AUTH=PLAIN IDLE ACL ACL2=UNION STARTTLS CAPABILITY OK CAPABILITY completed LOGOUT * BYE Courier-IMAP server shutting down LOGOUT OK LOGOUT completed Connection to host lost.

Example testing the connection to POP3 (port 110):$ telnet example.com 110 +OK Hello there. <5277.1488453319@localhost.localdomain> QUIT +OK Better luck next time. Connection to host lost.

PuTTY - Unencrypted Only

PuTTY is a popular terminal program used primarily under Windows, but also available under Linux. Although it supports an encrypted ssh mode, this is not compatible with the encryption on the mail ports.

If, for some reason, you have an aversion to telnet or just really love PuTTY, telnet mode is supported. To use putty in telnet mode, there are two options:
putty telnet:<address>:<port>
or
putty -telnet <address> <port>

For the SMTP example from telnet above: $ putty telnet:mail.example.com:25 $
The interaction then occurs in a separate PuTTY terminal:220 mail.example.com ESMTP Postfix (Ubuntu) EHLO example.com 250-example.com ...

openssl - Secure Connections

openssl supports both STARTTLS used on the default ports to upgrade an unencrypted connection to an encrypted connection (SMTP: 25, IMAP: 143, POP3: 110), and client SSL used on the encrypted ports (SMTPS: 465, IMAPS: 993, POP3S: 995).

Under Linux you may have to install the package. This will be
sudo apt install openssl
on most Linux systems.

Under Windows, the simplest way to get openssl is via Cygwin - at setup, add the openssl package to the default installation. The setup should install a Linux-like shell - you enter the openssl commands at that Linux-like shell rather than the usual DOS Prompt.

One can get OpenSSL for Windows, but I currently don't trust many of the sites offering binaries - you're on your own if you wish to pursue that.

Other MTAs will expect to port 25 to be open on your MTA, so testing SMTP/STARTTLS on port 25 should work. Note below that smtp is lowercase: openssl is case sensitive and expects the protocol in lowercase. In the examples, user input is bold, the shell prompt is "$": $ openssl s_client -connect mail.example.com:25 -starttls smtp CONNECTED(00000003) depth=2 C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA ... <Information on the certificate chain> ... ... <Information on the handshake/session> ... --- 250 DSN EHLO mail.example.com 250-mail.example.com 250-PIPELINING 250-SIZE 10240000 250-ETRN 250-AUTH DIGEST-MD5 CRAM-MD5 PLAIN LOGIN 250-ENHANCEDSTATUSCODES 250-8BITMIME 250 DSN QUIT DONE

Example testing IMAPS on port 993, without STARTTLS: $ openssl s_client -connect mail.example.com:993 CONNECTED(00000003) ... <Information on the certificate chain> ... ... <Information on the handshake/session> ... * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA AUTH=CRAM-MD5 AUTH=CRAM-SHA1 AUTH=CRAM-SHA256 AUTH=PLAIN IDLE ACL ACL2=UNION] Courier-IMAP ready. Copyright 1998-2015 Double Precision, Inc. See COPYING for distribution information. QUIT DONE

Authentication

A common mistake is assuming that your Linux user-base is the same as your email user-base - this is often not the case. For example, the plesk control panel on my VPS by default allows you to set up email accounts entirely independent of user login accounts. Conversely, under Windows, LDAP is configured to use the same logins.

Usually you can enable plain text logins when the connection is encrypted, so you can test a username/password pair when you connect using openssl. For both SMTP and IMAP, "plain" is actually base64 encoding. The following will work under Linux and Cygwin, substituting your login info for myname@myserver.com and mY-pasSwoRd, and of course your SMTP/IMAP server for myserver.com in the openssl command: $ ######## SMTP Login Example ######## $ # Get the username encoded in base64 $ echo -en "myname@myserver.com" | base64 bXluYW1lQG15c2VydmVyLmNvbQ== $ # Get the password encoded in base64 $ echo -en "mY-pasSwoRd" | base64 bVktcGFzU3dvUmQ= $ # Now login using those two strings in that order $ openssl sclient -connect myserver.com:25 -starttls smtp CONNECTED(00000003) ... 250 DSN AUTH LOGIN 334 VXNlcm5hbWU6 bXluYW1lQG15c2VydmVyLmNvbQ== 334 UGFzc3dvcmQ6 bVktcGFzU3dvUmQ= 235 2.7.0 Authentication successful QUIT DONE $ # In case you are wondering about the lines beginning with "334": $ echo -en "VXNlcm5hbWU6" | base64 --decode Username: $ echo -en "UGFzc3dvcmQ6" | base64 --decode Password: $ $ ######## IMAP Login Example ######## $ # Get the IMAP-login (username and password together) encoded in base64 $ echo -en "\0myname@myserver.com\0mY-pasSwoRd" | base64 AG15bmFtZUBteXNlcnZlci5jb20AbVktcGFzU3dvUmQ= $ # Now connect and login using the string we got $ openssl sclient -connect myserver.com:993 ... * OK [CAPABILITY IMAP4rev1 ... a001 AUTHENTICATE PLAIN + AG15bmFtZUBteXNlcnZlci5jb20AbVktcGFzU3dvUmQ= a001 OK LOGIN Ok. QUIT DONE

Checking DNS Entries

Before checking your DNS entries, you may want to review part 2 of this series, Subdomains and DNS.

There are two basic tools for examining DNS records that are essentially equivalent: nslookup and dig.

nslookup

nslookup is an old program for looking up DNS entries. There are different variants available under Linux and Windows, but they have the same basic purpose, have a similar syntax and are installed by default. If you enter nslookup without parameters, it will enter interactive mode. I will not be covering interactive mode here.

The most essential DNS entry is the MX entry: this tells external servers where they should send email. For example, if you receive mail for example.com, the lookup syntax would be the same under Windows and Linux, though the response will look slightly different:$ nslookup -query=MX example.com

If this returns the wrong entry, I would try again using the host itself as the DNS server:$ nslookup -query=MX example.com example.com

If it shows the wrong server, too, I would check my configuration (in my case, the panel in my VPS).

If it does return the correct entry, then I know I have a propagation error: my settings aren't being propagated to a higher level DNS server. On my server, this also means checking the panel: propagating to my primary DNS server is not automatic, but can be initiated via the panel. This may or may not be the case on your server(s).

Looking up the TXT records of example.com:$ nslookup -query=TXT example.com Server: local-dns-server.xyz Address: 111.222.123.234#53 Non-authoritative answer: example.com text = "v=spf1 +a +mx -all" example.com text = "\"mailconf=https://example.com/mail/config-v1.1.xml\"" example.com text = "google-site-verification=qwertzuiop123456789"
Note that in this example response, there are two answers that are interesting here: the SPF record and the mailconf record. The SPF record lets other servers know who is allowed to send mail from example.com (unless you're using a huge server, it should be identical to that above). The mailconf record can redirect where Thunderbird clients look for their autoconfiguration file and is optional.

The google-site-verification record is not relevant for email: it is an entry a Google user adds to verify to Google that they own the site (look up Google WebMaster Tools).

Finally, SRV records do not propagate, so one must always specify the host itself as the DNS server. Here is an example looking up the IMAP / SRV record of softwareschmiede-herndon.de:$ nslookup -query=SRV _imap.tcp.softwareschmiede-herndon.de softwareschmiede-herndon.de Server: softwareschmied-herndon.de Address: 5.35.246.86 _imap.tcp.softwareschmiede-herndon.de SRV service location: priority = 0 weight = 10 port = 143 svr hostname = softwareschmiede-herndon.de softwareschmiede-herndon.de nameserver = ns2.hans.hosteurope.de softwareschmiede-herndon.de nameserver = lvps5-35-246-86.dedicated.hosteurope.de softwareschmiede-herndon.de internet address = 5.35.246.86 lvps5-35-246-86.dedicated.hosteurope.de internet address = 5.35.246.86

dig

dig is the same basic functionality as nslookup, with some minor differences:

dig uses the OS's resolver libraries, nslookup uses its own internal libraries
nslookup was deprecated for a time in favor of dig - it is no longer deprecated.
dig will automatically use the host itself as DNS server when looking up SRV records.
dig output is generally easier to parse and manipulate if you are writing a script.
dig is not installed by default under Windows - if you need it instead of nslookup, you should probably install Cygwin and use it there.

Some examples:$ dig MX example.com $ dig TXT example.com $ dig SRV _imap._tcp.example.com

Cartoon GIF of a man setting a briefcase down, it unfolding to a flying car, him hopping in and flying away

Autoconfiguration

If Thunderbird or Outlook fails to configure correctly for your server, then the simplest test is to retrieve the XML document from your server. Open your browser and enter the URL in the address bar. Some browsers don't present XML well, especially the MS autodiscovery XML - you can then press Ctrl-U to examine the source.

My own server as an example for Thunderbird:
http://softwareschmiede-herndon.de/mail/config-v1.1.xml

My own server as an example for Outlook:
www.softwareschmiede-herndon.de/autodiscover/autodiscover.xml?<EMailAddress>wdh@softwareschmiede-herndon.de</EMailAddress>

For Programmers

It is quite possible to download and compile Thunderbird or other Open Source clients to see what is happening under the hood client-side. In fact, I did so to debug problems I was having with Thunderbird autoconfiguration. If you have programming experience, you may want to do likewise to debug client problems.

Some quick notes for people who want to debug in Thunderbird:

Here are quick instructions on how to build Thunderbird
At least for autoconfiguration, the real action occurs in JavaScript. If you've installed as recommended, you'll find the files at: ~/src/comm-central/mailnews/base/prefs/content
I did not find the log file for debug output, I ended up just adding additional information to the exception, which you see in the output when Thunderbird is started from a shell. I recommend redirecting stderr/stdout to a log file of your choice.

The author looking through a magnifying glass

In Closing

The aim of this post has been to provide you with the tools you need to find whatever is causing your email server problems. I hope it has succeeded.

The final blog post is done! Hooray! What I thought would be a single longish blog post has ended up being six longish blog posts spread out over months. I hope it has been helpful to you.

If you have corrections, updates, or suggestions for improvements, please contact me!

Thank you for reading!