Recently I've mostly been working with PHP, and debugging my programs on a server, using Eclipse and XDebug remote debugging when it needed to be step-by-step. This has not always worked well, but it worked well enough for most purposes.

Usually I just wrote the code, uploaded it, tried it out, fixed problems. Only occasionally would I need to open Eclipse and do step-by-step debugging. When I opened Eclipse, all of the windows to the source code would be blank, and I had to close them all, open the remote-access project, and then re-open the source. It was a pain, but doable.

Then, for some reason Eclipse and XDebug decided that they weren't going to handle breakpoints properly on my latest Joomla-based project (Joomla is a PHP-based CMS).

And then Eclipse refused to terminate - it had to be killed using the Task Manager.

After several rounds of dealing with that, I figured out how to fix the termination problem using the "-clean" and "-clearPersistedState" options at startup. But it still didn't want to do step-by-step debugging.

Okay. It is probably some interaction between the remote access and Joomla's massive number of files. I'll have to install and work it locally. Originally that was a problem, but I managed to get Microsoft SQL Server working with PHP locally a couple of months ago, so this should be no biggie. Right?

Joomla installation is simple - just unzip the files and open the root folder in your browser. Enter some data - especially database login credentials - and you're off to the races.

Except this time, it returns to the final "these are the options you chose" screen, before the "success" screen.

No error message. No red markings saying "this option is bad", nothing. It just returns.

After several rounds of retrying with varied options, I tried duckduckgoing the problem (Duckduckgo is like Google, but without tracking you and biasing your results). No one seemed to have my specific problem: most people were using Linux instead of Windows. And when they were using Windows, they used XAMP (a package containing everything one needs for a modern webserver: Apache WebServer, Tomcat, Java, MySQL). I hadn't installed that because when I first tried it, I didn't have enough space, and because I needed some feature of Microsoft's IIS at the time.

So... being the stubborn SOB I am, I decided to debug the problem. After a false start or two, I got Eclipse and XDebug to play nice locally, and started the Joomla installation. Boom: before the first Joomla screen, it gives me two error messages in the browser: some MySQL error, followed by an error about sending headers twice. sigh.

So, why is it even trying MySQL? I try debugging into it, to find out why it's calling MySQL, which I don't have installed. It turns out it simply tries it as its first default. Hmmm. I change the default to SQLSRV (which I think I have), but that fails for some reason - it later turns out that the Express version is handled differently, but at this point, I decide that maybe, just maybe, the second error message is the real problem.

I deactivate XDebug, and the error messages go away, we're back to the previous situation. I re-activate, the error messages are back. Some duckduckgoing, and I find that the second header problem occurs when one attempts to setup output and there's already output - like an error screen. Looking through the XDebug options I find that there's an option I've been activating that sends the error messages straight to the screen. I deactivate this, and vóila!, I get my Joomla installation screen. MySQL fails silently, it recovers correctly and then proceeds to try other options until it finds the one that works.

Okay. So, I proceed to install Joomla with the debugger in the background, and ... it returns to the final screen, just as before. The debugger doesn't automatically stop at an error. Of course not. That would make things too easy.

So, hours of step-by-step debugging later, using all the tricks I know to debug through the forest that is Joomla, I find that it is throwing an exception in JArrayHelper::toObject, trying to create an object of type "stdclass". The header is:

public static function toObject(&$array, $class = 'stdClass', $recursive = true)

Anyone notice anything?

Take your time.

Yeah. So, I set a breakpoint at the function and ... no, that doesn't work at all: this thing is called everywhere!

Okay. I go and modify toObject(), telling it to do something if the class is all lowercase, and then setting a breakpoint on that "do something" line. This works: it stops right where it should.

Looking up the stack, the call only provides one parameter, so $class should be 'stdClass'. But it's 'stdclass'. That should not happen - it looks like a problem in PHP.

I correct it to 'stdClass' and let it proceed: I get the "success" screen from Joomla installation.

Did I mention that I'm stubborn? There's thousands of Joomla programmers, some of them must have hit this rock and given up - installed XAMP, worked around it, or gone and joined a monastery.

I set about figuring out what PHP call provokes the problem, and it turns out that the last call before 'stdClass' becomes 'stdclass' is a call to "$query->execute()" - in other words, to my non-standard database. So probably not PHP directly - probably the PHP database driver for Microsoft SQL Server. The one that everyone seems to complain about, just before telling you to use MySQL instead.

I look for the most recent version, and: apparently the drivers are offered for download by Microsoft, and it tells you that the most recent version is 4.3 and only supported for Windows 8 and later (I'm still on Windows 7). I decide to download and try it anyway. Except that the download only gives me 4.0 as an option. I double-check. Yep. The Microsoft page with the description telling me about 4.3 links to a page where 4.3 isn't available. A bit of work later, I deduce that I'm using the latest version of the 4.0 driver already.

A bit more searching later, and I find the one 4.3 'native' binary (shouldn't there be two? The PDO and the native version?) - I install the native binary anyway, but it doesn't work. Of course. PHP thinks there's no database access at all.

This is a challenge. I find the sources for the driver and download them, but they don't come with instructions. And don't compile properly if I just try the obvious.

Okay. The driver is a PHP extension, I'll lookup how to build PHP extensions in general - that's likely a clue.

The recommendation is, that one build PHP before one builds extensions. And for my version, PHP 7.0, if you are building under Windows instead of Linux, you will need Microsoft Visual C/C++ 2015 or later (Note: I already have two previous versions of Visual Studio cluttering my hard drive, I don't want a third). Why does it need Microsoft tools (as opposed to MingW), and why that version? Apparently because it was developed by people at Microsoft, who are familiar with SQL Server, and of course they use the latest and greatest.

Did I mention that I'm stubborn?

After some modifications to the include files and two source files, I get it through the compile phase of Visual C/C++ 2010, but the link fails with an unknown link option "-FitObjData".

One of the things I had to change to get it to compile, was a misuse of the ALLOCA_FLAG() macro - something that could overwrite things on the stack and cause random problems precisely like the one observed. I should probably notify the PHP developers about that.

So, none of the build files passes this "-FitObjData" anywhere. It must be an implicit flag of some sort. A search online shows first that this occurs when you use newer libraries with an older linker. Recommended solution is to use the same compiler for both. Hmmm. The libraries are provided, and it would take days or weeks to get all of their sources and recompile them all for this. A bit more searching, and somewhere, someone mentions "inlining" in connection with this linker problem, and that they gave up, too. Aha! Inlining is an optimization. If I set it up for debugging, then optimizations should be turned off, and maybe that will get rid of the problem.

It does. But now it's having trouble with doubly-defined functions and missing functions, related to vsprintf(). Oh, no. I've run into this kind of problem before, when I was working on embedded software. I ended up rolling my own vsprintf() equivalent, because all the different vsprintf()s are incompatible depending on your compiler options and dangerous to mix-and-match. It is time to give up on this path.

So, I go and install MySQL? You don't know me very well.

No, I'm going to compile PHP, even if I have to bite the bullet and install Visual Studio 2015. A couple hours later, I've signed up for some free developer stuff from Microsoft and have started the install for the free version of Visual Studio 2015. The installation takes a couple hours, and the installation of the Windows 10 SDK fails, but the rest appears to work.

So, this is the recommended toolset with everything setup exactly as recommended in the instructions. Everything should just work, right?

Ha.

A clean build of the default configuration tells me: "fatal error C1007: unrecognized flag '-FitObjData' in 'p2'", just as before. That isn't supposed to happen - I have the new tools! Well, what worked once may work again - activate debug build!

The debug version tells me: lots of trouble with missing vsprintf() functions again! Verfluchte ... Wait a sec. It's "php7ts_debug.dll" is missing these functions - that's the thread safe version. Despite thread-safe being, well, safer, I know that the standard version is non-thread-safe. Time to rebuild!

The non-thread safe and debug version tells me: vsprintf() functions are still missing.

It's been over two days now. I've got things to get done. I don't have time for this crap anymore. I'm going to install Joomla using the debugger to set "stdclass to "stdClass" and move on. For now.

This is the final post in the now six part series "Serving Email". In this post we will explore how to figure out what's wrong when your email server isn't working properly.

This series:

• Part 1: The Pieces of the Puzzle
• Part 2: Subdomains and DNS
• Part 3: Encryption and SSL Certificates
• Part 4: The Email Server(s)
• Part 5: Autoconfiguration
• Part 6: Debugging Email Problems <- You are here

Where to Start

There are several parts to test when testing an email server. At the most basic level:

• Can you send emails from the server?
• Can you receive emails sent to the server?

Each of these can be broken down further:

• Can the service be located: is the meta information (DNS, et al.) about the service correct?
• Can the appropriate service be connected to on the appropriate socket?
• Is the SSL-certificate correct? (correct name? correct dates? etc.)?
• Does a secure connection occur?
• Is the authentication information correct?
• Does autoconfiguration set things correctly?

Also: is the MTA (SMTP/outgoing server) storing the emails in the same format and location that the IMAP/POP3/incoming server is expecting?

This is my process (YMMV):

• Set up Thunderbird as a client to the mail server
  • if it autoconfigures, I verify the parameters
  • if needed correct the parameters manually
• Send myself an email (server -> itself)
• Once that works, I try sending an email from another server
• Once that works, I try sending an email to another server
• Verify that autoconfiguration works for Thunderbird, Outlook
• Test Configuration Profile for iPhone

I leave fixing autoconfiguration for last, even though it occurs first: I can always configure manually until it works. At any other point where it fails, I check likely failure points.

If the connection fails it is usually obvious in the test phase of your email client configuration. I try manually connecting using the appropriate tool (see section below) and if that succeeds, manually authenticating. Possible causes are misconfiguration of the server and firewall interference.

More commonly (in my experience) the connection works, but it fails to send or receive mails. The most common cause for that is misconfigured DNS entries.

Testing Connections

If you're having trouble receiving emails or connecting as a client, you'll want to test if the appropriate port for the protocol is open and you can connect correctly.

I usually test the unencrypted connection before testing the encrypted connection - you can go straight to the encrypted connection (below), but if the server supports the unencrypted ports (STARTTLS starts out unencrypted) I like to use telnet to make sure that the ports are open and the server responding.

There are several programs that you can use to test the connections to a selected port that you need for testing.

telnet - Unencrypted Only

telnet is an ancient program (I used it in college over 30 years ago) that only gives you a simple unencrypted TCP/IP connection. It is available under both Linux and Windows by default.

Example testing the connection to SMTP (port 25):
$ telnet mail.example.com 25 220 mail.example.com ESMTP Postfix (Ubuntu) EHLO example.com 250-example.com 250-PIPELINING 250-SIZE 10240000 250-ETRN 250-STARTTLS 250-AUTH DIGEST-MD5 CRAM-MD5 PLAIN LOGIN 250-ENHANCEDSTATUSCODES 250-8BITMIME 250 DSN QUIT 221 2.0.0 Bye   Connection to host lost.
Example testing the connection to IMAP (port 143):
$ telnet example.com 143 * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA AUTH=CRAM-MD5 AUTH=CRAM-SHA1 AUTH=CRAM-SHA256 AUTH=PLAIN IDLE ACL ACL2=UNION STARTTLS] Courier-IMAP ready. Copyright 1998-2015 Double Precision, Inc. See COPYING for distribution information. CAPABILITY * CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA AUTH=CRAM-MD5 AUTH=CRAM-SHA1 AUTH=CRAM-SHA256 AUTH=PLAIN IDLE ACL ACL2=UNION STARTTLS CAPABILITY OK CAPABILITY completed LOGOUT * BYE Courier-IMAP server shutting down LOGOUT OK LOGOUT completed   Connection to host lost.

Example testing the connection to POP3 (port 110):
$ telnet example.com 110 +OK Hello there. <5277.1488453319@localhost.localdomain> QUIT +OK Better luck next time.   Connection to host lost.

PuTTY - Unencrypted Only

PuTTY is a popular terminal program used primarily under Windows, but also available under Linux. Although it supports an encrypted ssh mode, this is not compatible with the encryption on the mail ports.

If, for some reason, you have an aversion to telnet or just really love PuTTY, telnet mode is supported. To use putty in telnet mode, there are two options:
putty telnet:<address>:<port>
or
putty -telnet <address> <port>

For the SMTP example from telnet above:
$ putty telnet:mail.example.com:25 $
The interaction then occurs in a separate PuTTY terminal:
220 mail.example.com ESMTP Postfix (Ubuntu) EHLO example.com 250-example.com ...

openssl - Secure Connections

openssl supports both STARTTLS used on the default ports to upgrade an unencrypted connection to an encrypted connection (SMTP: 25, IMAP: 143, POP3: 110), and client SSL used on the encrypted ports (SMTPS: 465, IMAPS: 993, POP3S: 995).

Under Linux you may have to install the package. This will be
sudo apt install openssl
on most Linux systems.

Under Windows, the simplest way to get openssl is via Cygwin - at setup, add the openssl package to the default installation. The setup should install a Linux-like shell - you enter the openssl commands at that Linux-like shell rather than the usual DOS Prompt.

One can get OpenSSL for Windows, but I currently don't trust many of the sites offering binaries - you're on your own if you wish to pursue that.

Other MTAs will expect to port 25 to be open on your MTA, so testing SMTP/STARTTLS on port 25 should work. Note below that smtp is lowercase: openssl is case sensitive and expects the protocol in lowercase. In the examples, user input is bold, the shell prompt is "$":
$ openssl s_client -connect mail.example.com:25 -starttls smtp CONNECTED(00000003) depth=2 C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA ... <Information on the certificate chain> ... ... <Information on the handshake/session> ... --- 250 DSN EHLO mail.example.com 250-mail.example.com 250-PIPELINING 250-SIZE 10240000 250-ETRN 250-AUTH DIGEST-MD5 CRAM-MD5 PLAIN LOGIN 250-ENHANCEDSTATUSCODES 250-8BITMIME 250 DSN QUIT DONE

Example testing IMAPS on port 993, without STARTTLS:
$ openssl s_client -connect mail.example.com:993 CONNECTED(00000003) ... <Information on the certificate chain> ... ... <Information on the handshake/session> ... * OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA AUTH=CRAM-MD5 AUTH=CRAM-SHA1 AUTH=CRAM-SHA256 AUTH=PLAIN IDLE ACL ACL2=UNION] Courier-IMAP ready. Copyright 1998-2015 Double Precision, Inc. See COPYING for distribution information. QUIT DONE

Authentication

A common mistake is assuming that your Linux user-base is the same as your email user-base - this is often not the case. For example, the plesk control panel on my VPS by default allows you to set up email accounts entirely independent of user login accounts. Conversely, under Windows, LDAP is configured to use the same logins.

Usually you can enable plain text logins when the connection is encrypted, so you can test a username/password pair when you connect using openssl. For both SMTP and IMAP, "plain" is actually base64 encoding. The following will work under Linux and Cygwin, substituting your login info for myname@myserver.com and mY-pasSwoRd, and of course your SMTP/IMAP server for myserver.com in the openssl command:
$ ######## SMTP Login Example ######## $ # Get the username encoded in base64 $ echo -en "myname@myserver.com" | base64 bXluYW1lQG15c2VydmVyLmNvbQ==   $ # Get the password encoded in base64 $ echo -en "mY-pasSwoRd" | base64 bVktcGFzU3dvUmQ=   $ # Now login using those two strings in that order $ openssl sclient -connect myserver.com:25 -starttls smtp CONNECTED(00000003) ... 250 DSN AUTH LOGIN 334 VXNlcm5hbWU6 bXluYW1lQG15c2VydmVyLmNvbQ== 334 UGFzc3dvcmQ6 bVktcGFzU3dvUmQ= 235 2.7.0 Authentication successful QUIT DONE $ # In case you are wondering about the lines beginning with "334": $ echo -en "VXNlcm5hbWU6" | base64 --decode Username: $ echo -en "UGFzc3dvcmQ6" | base64 --decode Password: $ $ ######## IMAP Login Example ######## $ # Get the IMAP-login (username and password together) encoded in base64 $ echo -en "\0myname@myserver.com\0mY-pasSwoRd" | base64 AG15bmFtZUBteXNlcnZlci5jb20AbVktcGFzU3dvUmQ=   $ # Now connect and login using the string we got $ openssl sclient -connect myserver.com:993 ... * OK [CAPABILITY IMAP4rev1 ... a001 AUTHENTICATE PLAIN + AG15bmFtZUBteXNlcnZlci5jb20AbVktcGFzU3dvUmQ= a001 OK LOGIN Ok. QUIT DONE

Checking DNS Entries

Before checking your DNS entries, you may want to review part 2 of this series, Subdomains and DNS.

There are two basic tools for examining DNS records that are essentially equivalent: nslookup and dig.

nslookup

nslookup is an old program for looking up DNS entries. There are different variants available under Linux and Windows, but they have the same basic purpose, have a similar syntax and are installed by default. If you enter nslookup without parameters, it will enter interactive mode. I will not be covering interactive mode here.

The most essential DNS entry is the MX entry: this tells external servers where they should send email. For example, if you receive mail for example.com, the lookup syntax would be the same under Windows and Linux, though the response will look slightly different:
$ nslookup -query=MX example.com

If this returns the wrong entry, I would try again using the host itself as the DNS server:
$ nslookup -query=MX example.com example.com

If it shows the wrong server, too, I would check my configuration (in my case, the panel in my VPS).

If it does return the correct entry, then I know I have a propagation error: my settings aren't being propagated to a higher level DNS server. On my server, this also means checking the panel: propagating to my primary DNS server is not automatic, but can be initiated via the panel. This may or may not be the case on your server(s).

Looking up the TXT records of example.com:
$ nslookup -query=TXT example.com Server: local-dns-server.xyz Address: 111.222.123.234#53   Non-authoritative answer: example.com text = "v=spf1 +a +mx -all" example.com text = "\"mailconf=https://example.com/mail/config-v1.1.xml\"" example.com text = "google-site-verification=qwertzuiop123456789"
Note that in this example response, there are two answers that are interesting here: the SPF record and the mailconf record. The SPF record lets other servers know who is allowed to send mail from example.com (unless you're using a huge server, it should be identical to that above). The mailconf record can redirect where Thunderbird clients look for their autoconfiguration file and is optional.

The google-site-verification record is not relevant for email: it is an entry a Google user adds to verify to Google that they own the site (look up Google WebMaster Tools).

Finally, SRV records do not propagate, so one must always specify the host itself as the DNS server. Here is an example looking up the IMAP / SRV record of softwareschmiede-herndon.de:
$ nslookup -query=SRV _imap.tcp.softwareschmiede-herndon.de softwareschmiede-herndon.de Server: softwareschmied-herndon.de Address: 5.35.246.86   _imap.tcp.softwareschmiede-herndon.de SRV service location: priority = 0 weight = 10 port = 143 svr hostname = softwareschmiede-herndon.de softwareschmiede-herndon.de nameserver = ns2.hans.hosteurope.de softwareschmiede-herndon.de nameserver = lvps5-35-246-86.dedicated.hosteurope.de softwareschmiede-herndon.de internet address = 5.35.246.86 lvps5-35-246-86.dedicated.hosteurope.de internet address = 5.35.246.86


dig

dig is the same basic functionality as nslookup, with some minor differences:

• dig uses the OS's resolver libraries, nslookup uses its own internal libraries
• nslookup was deprecated for a time in favor of dig - it is no longer deprecated.
• dig will automatically use the host itself as DNS server when looking up SRV records.
• dig output is generally easier to parse and manipulate if you are writing a script.
• dig is not installed by default under Windows - if you need it instead of nslookup, you should probably install Cygwin and use it there.

Some examples:
$ dig MX example.com $ dig TXT example.com $ dig SRV _imap._tcp.example.com

Autoconfiguration

If Thunderbird or Outlook fails to configure correctly for your server, then the simplest test is to retrieve the XML document from your server. Open your browser and enter the URL in the address bar. Some browsers don't present XML well, especially the MS autodiscovery XML - you can then press Ctrl-U to examine the source.

My own server as an example for Thunderbird:
http://softwareschmiede-herndon.de/mail/config-v1.1.xml

My own server as an example for Outlook:
www.softwareschmiede-herndon.de/autodiscover/autodiscover.xml?<EMailAddress>wdh@softwareschmiede-herndon.de</EMailAddress>

For Programmers

It is quite possible to download and compile Thunderbird or other Open Source clients to see what is happening under the hood client-side. In fact, I did so to debug problems I was having with Thunderbird autoconfiguration. If you have programming experience, you may want to do likewise to debug client problems.

Some quick notes for people who want to debug in Thunderbird:

• Here are quick instructions on how to build Thunderbird
• At least for autoconfiguration, the real action occurs in JavaScript. If you've installed as recommended, you'll find the files at: ~/src/comm-central/mailnews/base/prefs/content
• I did not find the log file for debug output, I ended up just adding additional information to the exception, which you see in the output when Thunderbird is started from a shell. I recommend redirecting stderr/stdout to a log file of your choice.

In Closing

The aim of this post has been to provide you with the tools you need to find whatever is causing your email server problems. I hope it has succeeded.

The final blog post is done! Hooray! What I thought would be a single longish blog post has ended up being six longish blog posts spread out over months. I hope it has been helpful to you.

If you have corrections, updates, or suggestions for improvements, please contact me!

Thank you for reading!

Copyright © 2017 William David Herndon. All rights reserved.

This series:

• Part 1: The Pieces of the Puzzle
• Part 2: Subdomains and DNS
• Part 3: Encryption and SSL Certificates
• Part 4: The Email Server(s)
• Part 5: Autoconfiguration
• Part 6: Debugging Email Problems <- You are here

This is part five of a six part series. In this part we will cover setting up Autoconfiguration for Email Clients.

This series:

• Part 1: The Pieces of the Puzzle
• Part 2: Subdomains and DNS
• Part 3: Encryption and SSL Certificates
• Part 4: The Email Server(s)
• Part 5: Autoconfiguration <- You are here
• Part 6: Debugging Email Servers

This post in the series will cover Autoconfiguration: setting things up so that clients can automatically get the right server settings, bothering the user as little as possible. Preferably they should only need to give the Email Client their email address and password, and it should just work.

There are two major clients that support autoconfiguration: Microsoft Outlook (Microsoft calls it "autodiscovery") and Mozilla Thunderbird. The Apple iPhone does not support autoconfiguration, but it does support importing what they call a "configuration profile". I will be covering all three of these and some other issues surrounding configuration of other clients.

Practically this means that you, as an operator of an Email Server, will want to implement everything you can to minimize user support requests, unless by some lucky chance you're users are limited to some subset of the email clients.

Mozilla Thunderbird

Autoconfiguration

Quick Start

• Right-click and download this XML-file
• Modify it to fit your server settings, deleting sections that don't apply.
• Then, if you can, upload it to your server at this location:

<base-domain>/.well-known/autoconfig/mail/config-v1.1.xml

If your website redirects using ProxyPass / ProxyPassReverse, you can set it up to not redirect that one directory. An example from the Apache settings for this blog:
RewriteEngine On ProxyPass /.well-known ! ProxyPass / http://localhost:2368/ ProxyPassReverse / http://localhost:2368/

.well-known is a sub-directory that is also used for other files that are not directly connected with website content. For instance for verifying site ownership.

Alternatively, if it is not possible to make the file available at directory .well-known, but you can create a new subdomain, then create the subdomain autoconfig.<base-domain>, and place the configuration file at:

autoconfig.<base-domain>/mail/config-v1.1.xml

Short Explanation

Thunderbird will check the MX record in DNS of the host part of the email address (user@myhost.com). If it finds none, it will use the host part of the email address instead. Then it will try to download the file first from autoconfig.<base-address>/..., then from <base-address>.well-known/... location. I suggest the other way around, because a simple upload is usually easier than creating a new subdomain.

The XML configuration file then has all the information needed to configure the client: the servers, the protocols, the ports and some of the security options (other security options are negotiated at connection).

Original Documentation

• Here is the original documentation from Mozilla
• Here is a direct link to a description of the file format

Microsoft Outlook

Autodiscovery

Quick Start: but probably incomplete

• Right-click and download this XML-file
• Delete the <LoginName> sections.
• Modify it to fit your server settings
• Then upload it to this location:

<base-domain>/autodiscover/autodiscover.xml

Why it is probably incomplete: when the <LoginName> section is missing, Outlook will assume the login name is the username part of the email address without the domain-name (user@mydomain.com). Most modern Email Servers support multiple domains, and consequently require the full email address, with domain, for login.

The Thunderbird autoconfiguration handles this simply: %EMAILADDRESS% substitutes for the full email address. Microsoft Outlook, instead, passes the email address as a parameter, unavailable to non-executing files. If you want to get a complete fix, press on.

Not so Quick Start

In order to retrieve the email address, you need to execute code when Outlook retrieves the autodiscover.xml. If your website supports PHP, you may copy my solution and adjust as needed. If you have something other than PHP, then you will need to write your own code. My solution:

• Download this file
• Rename extension from .txt to .php
• Adjust the fields as needed for your email server.
• Upload it to the autodiscover directory
• Adjust your Apache configuration to retrieve autodiscover.php when autodiscover.xml is requested: RewriteEngine On RewriteRule autodiscover/autodiscover.xml /autodiscover/autodiscover.php

Short Explanation

When Outlook retrieves autodiscover.xml, it passes the emailaddress and other parameters in the POST data in XML format. POST data may be passed in one of two forms: one is in the address following a "?" - you can simply enter this in the browser - the other is embedded in the POST-exchange - the one that Outlook uses. The PHP I wrote checks both, so that I can test the results using a browser:

http://www.softwareschmiede-herndon.de/autodiscover/autodiscover.xml?<EMailAddress>test@example.com</EMailAddress>

Original Documentation

• Here is the original autodiscovery documentation from TechNet

Apple iPhone/iPad/iOS

Configuration Profile

Unless you have an Email Server large enough that Apple includes you in their database, there is no autoconfiguration for you. You have two options: your users enter all the server info by hand, or you give your user a "Configuration Profile", that makes the entries for them.

I've developed a PHP program that will generate a configuration profile and, when a user clicks download on their iPhone, it will start installing.

You can see the program in action here.

You can download the source here. It has the .txt extension so it is not interpreted, you will need to rename it to use it. Note that it is expecting two parameters, emailname and emailaddress. You can modify the program as needed for your website and direct your users to a webpage where they can use it.

Original Documentation

• Here is the original Configuration Profile Reference from Apple (search for "Email Payload")

Other Email Clients

I tested out other clients using an account at dojo-wehrda.de, which has an MX entry to softwareschmiede-herndon.de and a correct Thunderbird config file in .well-known, and a correctly functioning autodiscovery.xml.

So far none have done correct autoconfiguration, despite the standards for DNS SRV entries and the autoconfiguration info presented here. If you have an Email Client that you're fond of, and they don't support autoconfiguration for smaller servers, please inform them about it.

I started out this series by showing screenshots of K-9 Mail and how it should work. Since I use K-9 for multiple email addresses, and some of them automatically configured, I thought they must have some form of autoconfiguration I could use. I was wrong: they have a database of all the major providers and how they should be configured. For smaller Email Servers like mine it will simply throw you into advanced configuration. It is an Open Source project: I intend to talk to them about this and get it fixed or fix it myself.

Next I tried MailDroid, a commercial Android App, with a free ad-based version. It simply said it could not locate the server. And threw me into the incoming/outgoing server dialog. Oh, well.

BlueMail, another commercial Android App with a free version, seemed to work - it didn't complain about anything, and even showed an email from itself congratulating me on my choice. After waiting several minutes for an update that never came, I went to the advanced settings, where I discovered that it had ignored the DNS and the autoconfiguration files and was trying to synchronize via webmail.dojo-wehrda.de. Fail.

Claws-Mail, the Linux-based Email client, fared a little bit better: at first it tried to setup a POP3 account on my local machine, despite my email address. However when I changed the setting to IMAP and pressed [Auto-configure], it correctly found softwareschmiede-herndon.de and its settings. It then proceeded to SMTP, and again had the local machine set as server, without an [Auto-configure] button - to correctly configure outgoing, I would have to enter everything by hand.

The hardest fail was TheBat, a very old Windows-based client - I used it for a number of years with Windows 95. It is a favorite of old programmers. Despite the version being from this year, it failed hard: it did not find softwareschmiede-herndon.de. Clicking the [Testen!] button on the IMAP page with the defaults did not react - no success or error message - it should have told me that the certificate presented was for the wrong server (the certificate is for softwareschmiede-herndon.de, of course). Proceeding to the SMTP page, clicking on [Testen!] caused the dialog to freeze up, so that I ended up using the task manager to kill the process. I used TheBat 32-bit version on Windows 7 Professional, 32-Bit version.

Notes

Regarding DNS and Clients

If you've been following along, you'll know the importance of DNS records regarding Email Server configuration - post #2 was mostly about DNS. Funny thing: a lot of clients cannot access DNS at all, either because of limitations on their platform, or due to being blocked by a firewall. Mozilla Thunderbird skirts the problem by using an MX-lookup server available via HTTP(S).

In Closing

If you've been implementing as you follow along, you should have a functioning server that is easy for users to configure their client software for. If you don't, you'll be happy to know that I've decided to extend this series by yet one more post: Debugging Email Servers.

Thank you for reading my blog post!

Copyright © 2017 William David Herndon. All rights reserved.

This series:

• Part 1: The Pieces of the Puzzle
• Part 2: Subdomains and DNS
• Part 3: Encryption and SSL Certificates
• Part 4: The Email Server(s)
• Part 5: Autoconfiguration <- You are here
• Part 6: Debugging Email Servers

This is part four of a six part series. In this part we will cover the Email Server(s).

This series:

• Part 1: The Pieces of the Puzzle
• Part 2: Subdomains and DNS
• Part 3: Encryption and SSL Certificates
• Part 4: The Email Server(s) <- You are here
• Part 5: Autoconfiguration
• Part 6: Debugging Email Servers

In this post: I discuss the criteria for choosing your email server(s) and then link to the Wikipedia email server comparison table, where you can find email servers that match your criteria.

There may be up to three different email server programs, possibly on separate servers, or they may all be combined in one software. They are:

• The Message Transfer Agent (MTA), also called the Outgoing Server.
• The Incoming Server.
• The WebMail Server (optional)

This diagram is the most complex part of the post - don't let it scare you:

Things to note about the diagram:

• I made the SMTP/IMAP/POP3 arrows unidirectional to indicate which way the emails flow, even though there is bidirectional flow of information (for example to authenticate, or to delete emails).
• Each of the components in the Local Server can be separated out onto a different machine, despite being shown on one machine here.
• In concept, the MTA is relatively simple: it will check an incoming email's address against its user-base and either put it in Email Storage, or look for an appropriate MTA to pass it on to. All communication is via SMTP.
• The Email Storage is shared between the Incoming Server (MTA) and the Outgoing Server, so they must use the same format and synchronize properly, whether that storage is file(s) or a database or something else.
• The Incoming Server never actively sends email, it waits for a client to retrieve it via IMAP and/or POP3.
• Most WebMail Servers work just like an IMAP-based Email Client Software, except that the user accesses it through HTTP(S) and Browser Software.
• This diagram leaves out the authentication database - user/password information. The Incoming Server and Outgoing Server should also share this information.

I put the criteria roughly in the order I think of as most relevant and important, not the order they appear in the Wikipedia comparison.

Panel

If you have a panel, like Plesk or cPanel, then I recommend using their defaults: their choice is usually pretty flexible and you will get better support. Personally I have Plesk on Linux and use the defaults: Postfix for outgoing, Courier for incoming, and Horde for WebMail.

The Operating System: Linux, Windows, or MacOS

Not much to say, really: you probably already have your operating system. Linux is free. Windows will cost. Linux has the largest market share for Email Servers. MacOS the smallest. Almost all that support MacOS support Linux, too, since they're both Unix derivatives and Linux has the larger market.

Server Types

• Outgoing Server: SMTP - sends and receives Email from remote servers.
• Incoming Server: IMAP, POP3 - retrieve the email on the server for a client. IMAP keeps the email on the server. POP3 deletes it from the server after retrieval - the client must store it.
• WebMail - Access via a web browser.
• ActiveSync - this is a Microsoft protocol for synchronizing emails, contacts, calendars, tasks and notes. Supporting these is beyond the scope of this series, but may be something you want to consider.

WebMail is the simplest to separate out: there are specialized WebMail server programs that do nothing else and only need IMAP and SMTP access to work. I would not make it a criteria that WebMail be integrated with my other servers. Note also that the Wikipedia comparison list excludes WebMail only servers.

Your incoming and outgoing server can be independent, but if they are, you will still need them to access common storage and want them to access a common set of usernames/passwords, so in that case, pay attention to the storage and the authentication storage options: filesystem, database, LDAP, other.

Earlier people used to always retrieve their email and delete it from the server - space on the server was expensive and you had one computer. That is the POP3 model. Nowadays, server space is cheap and people want to read their email on their mobile, their tablet and their laptop, so the email needs to be on the server to be accessible. This is the IMAP model. For general customers I recommend supporting both - there are some old-school folk still around. If you have to drop one, drop POP3. Separate IMAP and POP3 servers are not practical.

Secure Connection

Unless you're doing an in-house mail system in a local network, secure connections are an absolute must, both for outgoing and incoming. The SSL feature is straightforward: it means the supported protocols appear on their alternate SSL ports, often signified by appending an "S" to the protocol name (SMTPS on port 587, IMAPS on port 993, POP3S on port 995). The "Opportunistic TLS" options (also called STARTTLS), SMTP over TLS and POP over TLS mean that a plain text connection is made over the old lower port number, and then the connection is upgraded to a secure connection. Some people prefer the former, so that no meta-data can be caught by eavesdroppers. Some prefer the latter, so that fewer ports are used. Most servers I've seen support both, letting the client decide.

Authentication

Authentication means asking for username/password, and is an absolute must. Whether or not encrypted passwords are supported doesn't seem to be in the Wikipedia comparison, probably because with an encrypted connection, password encryption is now unimportant.

SMTP Authentication is in the list: there are servers that allow sending of emails without asking for a username and password at all, even though that is very likely to get your server blacklisted these days.

If you're using separate ingoing and outgoing servers, be sure they can use the same authentication database (Filesystem/Database/LDAP).

I would ignore "POP before SMTP" - it is almost never used anymore. I would also ignore APOP, a specialized form of encrypted authentication for POP3 that is not relevant for most with the rise of MAPI.

Storage: Database, File System, Other

The storage system for incoming and outgoing must be compatible. A given if you're only using one program for both, but needs to be checked if you're splitting it up.

This may also be relevant to your backup system or to your personal preferences: some old school programmers may like to see actual files they can edit with a text editor; a database specialist might like to be able to mangle things with SQL.

Mature / Active Development

New software tends to be buggy, so mature is good. It's not on the comparison list, so I would google the software that makes it to the final rounds. I would only go "bleeding edge" with email that I don't care about.

I think that IPv6 support is a good indicator of how actively the project is being developed. Support of IPv6 is usually not important yet, but it will be, and so a software that does support it is forward looking. I would, again, google to verify in the final rounds.

Antispam Features

I could do a whole series on antispam features. That said, for small operations I don't think they're a killer criterion - I receive spam, my email client mostly sorts it into the spam bin. It's not been a problem for me so far. If you're customers are not tech-savvy, you may want one with integrated antivirus.

Other Criteria

• IMAP - IDLE - real time update to email clients, without continually refreshing. Maybe relevant if you have client software that supports this and people want it.
• NNTP - Network News Protocol - for news readers. I've never actually used this, but if you do, go for it.
• Sieve is a filtering language. Examples of use: auto-rejecting too large emails, pre-sorting spam, pre-sorting of email into other groups. I think this has lost a lot of importance, especially with the other antispam options.
• Upgrade Path - there are tools that can do IMAP backup and restore, so I wouldn't worry about that too much. If you're doing ActiveSync/WebDAV, that's a different can of worms.

Compare the Servers

Click here: The Wikipedia Comparison of Mail Servers

Take your time and compare.

Most of them are free, so if you're having trouble deciding, you can do a temporary install to test it out.

Installation

• If you're running one of the panels, then it's probably already installed, and you only need to activate it. Your panel will have documentation.
• If you're under Linux, many of these are available in the distribution channel - e.g. my synaptic package manager lists these packages from the comparison list: Citadel, Courier, Cyrus, Dovecot, Exim, OpenSMTPD, Postfix, qmail, sendmail.
• For other everything else: find their website and follow their instructions.

Do not forget the SSL Certificate!

Your package will generally have instructions on how to set the certificate to be used. Just don't forget it, otherwise your users will get nasty messages and may not be able to use your server at all.

Special note for Plesk users: Plesk will automatically use the default certificate for the email server. Plesk has documentation on how to bypass this if you need to.

In Closing

We're almost there! You should have a working email server now. All that's left is to set it up so that email clients can find the proper settings more easily - which is a lot more complicated than it should be, in my opinion.

Thank you for reading my blog post!

Copyright © 2017 William David Herndon. All rights reserved.

This series:

• Part 1: The Pieces of the Puzzle
• Part 2: Subdomains and DNS
• Part 3: Encryption and SSL Certificates
• Part 4: The Email Server(s) <- You are here
• Part 5: Autoconfiguration
• Part 6: Debugging Email Servers

This is part three of a six part series on setting up an email server and covers encryption and SSL Certificates*. This post is also useful for someone who just wants https support for their website.

This series:

• Part 1: The Pieces of the Puzzle
• Part 2: Subdomains and DNS
• Part 3: Encryption and SSL Certificates <- You are here
• Part 4: The Email Server(s)
• Part 5: Autoconfiguration
• Part 6: Debugging Email Servers

* Technically a TLS Certificate, a type of Public Key Certificate, but it is still normally referred to by the old name, "SSL Certificate".

What the Heck is an SSL Certificate?

An SSL certificate is used to verify that your browser, email program, or other program is communicating securely with the address in the certificate. It does this by combining the address, a public key and a little bit of other info, then having a trusted entity digitally sign that combination.

Your web-browser, email program or other program has a list of self-signed "root certificates" that it trusts. If a certificate from a website is signed by one of these, then it will trust that certificate. Not only that, it will also trust any chain (certificate signed by certificate signed by ... signed by root certificate) that end in a trusted root certificate. An important proviso, is that any parent certificates needs to be a certificate authority certificate - thus you can't sign someone else's certificate, just because you have a website certificate. There are other important provisos, but that's the basics.

Validation

Certificate Authorities are required to verify that the certificates they sign are who they purport to be. There are three levels of validation: domain validation, organizational validation, and extended validation.

Domain Validation

This lowest level of validation simply verifies that you have control of the site. There are 3 ways this is usually done:

• Respond to an email to the registered webmaster or postmaster of the site
• Place a file in a specific location on the site
• Make a DNS record (see previous post in this series)

You may need to do these in combination: e.g. place a file sent to the webmaster on your website, or make a DNS record specified in an email to the postmaster.

Organization Validation (or Business Validation)

Certificates with organizational validation fill in the "organization" field in the certificate. They usually cost significantly more than just domain validation certificates.

Proving that you are the organizational or business entity that you say you are usually involves sending copies of your letter of incorporation, organizational charter or other business documents and may take days for the review.

Extended Validation

Then there's "Extended Validation" (EV) that will turn the address bar or text green and may place the organization's name in the address bar. In this case, the business validation is more extensive and should ensure the customer that your business entity is "above board". Note that wildcard certificates are not possible with EV. EV certificates are very expensive.

I find it telling, that neither Amazon, nor ebay have adopted EV, though PayPal has.

What Certificate(s) will you need?

You can and should use the same certificate(s) for https and for your MTA (Mail Transfer Agent). You will need certificate support for each subdomain the MTA is using, whether that is the single domain solution (example.com), the single extra subdomain solution (mail.example.com), or a multi-subdomain solution (imap.example.com, pop3.example.com, smtp.example.com).

Note that because of inherent limitations in the mail protocols, your MTA has only one domain per protocol / IP-Address. This means only one SSL Certificate per protocol / IP-Address. This does not mean that it cannot handle emails from all your domains: it merely requires that the domain be explicit in the email username. For instance: if you have the domains example.com and example2.com on the same server / IP address, the email client setup for me@example2.com might set the server to mail.example.com and the user for login to me@example2.com. One could not set the user to just me, because it would not be able to differentiate between me@example.com and me@example2.com.

A wildcard certificate should work, be careful however: I ran into a hitch with a wildcard certificate I bought: "*.softwareschmiede-herndon.de" is not a valid certificate for "softwareschmiede-herndon.de" (it's missing the ".") - this was not an email specific problem. This can be solved by either always redirecting to "www." or by getting a certificate that explicitly covers the main domain as well.

You will not need certificates for autoconfiguration (autoconfig.example.com, autodiscover.example.com), though they don't hurt either.

If you already have the certificate(s) you need and they are installed for https, then you should continue with Part 4 when that is posted.

If you have certificate(s), but they are not installed for https, you can skip to installation.

Obtaining Certificates

If you don't have the certificate(s) you need yet, there are the following options:

• Self-signed certificates.
• Obtain from Let's Encrypt. These are free, but only good for three months at a time.
• Obtain from your webhoster: this is often the easiest (sometimes Let's Encrypt is actually easier).
• Obtain from a third party.

Self-signed Certificates

One can generate a certificate for oneself and there are several guides in the internet to generating them. I tried it, and I no longer consider this an option: your users will get nasty warnings when they visit your website or try to use your mail server. On some clients, I found no in-program way to circumvent the lack of trust, meaning that the user has to figure out how to install a trusted certificate on their platform.

If you choose this option, you are on your own.

Let's Encrypt!

Let's Encrypt (letsencrypt.org) is a "free, open, and automated Certificate Authority." These certificates only have domain validation and are only good for three months, but they are free.

How that works in specifics is different based on which panel you're using or if you're using a panel at all.

Verification Difficulties

If your Let's Encrypt verification does not work, then you probably have some sort of redirect going on, such that the verification files placed on the website are not found. For instance this blog uses ProxyPass to serve from ghost blogging software, which is on port 2368. Let's Encrypt! uses the subdirectory ".well-known" (which is also used for other purposes), so I added "ProxyPass /.well-known !" to my "Apache & Nginx Settings" in Plesk which now looks like this:

HTTP:

RewriteEngine On
ProxyPass /.well-known !
ProxyPass / http://localhost:2368/
ProxyPassReverse / http://localhost:2368/

HTTPS:

RewriteEngine On
ProxyPass /.well-known !
ProxyPass / http://localhost:2368/
ProxyPassReverse / http://localhost:2368/
RequestHeader set X-Forwarded-Proto "https"

If you are using a different panel or a different webserver, you will need to adjust according to that software.

Plesk Let's Encrypt! Plugin

Here is the Plesk extension for Let's Encrypt!. It is free. Login to Plesk, click on "Extensions", click the button "Add Extension" and upload the extension. Then start the extension by clicking on it. You will see a list of your domains. For most domains, simply click on the domain to install a "Let's Encrypt!" certificate.

cPanel Let's Encrypt! Plugin

I do not currently use cPanel, but here is the cPanel plugin. It is probably worth the $30: doing a hand verification yourself every couple months will get old real quick.

Do it yourself Let's Encrypt! Certificates

Go to "Getting Started" at letsencrypt.org and follow the instructions. Currently, this mostly mean using certbot. Certbot can be setup on the webserver (if you have command line access) to do automatic renewals. Or it can be run in manual mode if you don't have command line access.

SSL Certificate from your Webhoster?

If Let's Encrypt! is not for you - you need organizational validation, a wildcard certificate, or certificates that are valid for years - then the easiest route is almost certainly your webhoster: your webhoster will already have domain validation, may already have organizational validation through your business relationship with them and the process may or may not be integrated into your panel. They may also allow you to pay for the certificate monthly: be careful not to compare monthly prices to yearly prices!

A certificate from your webhoster will almost certainly be more expensive than from a 3rd party: the margins are thin and competition fierce in webhosting, so ISPs try to cash in on their "captive audience" once they have you. Fortunately, webhosters generally allow 3rd party certificates - they would lose many transfer customers if they didn't. So unless money is no object, I recommend shopping around.

SSL Certificate from a 3rd Party

If you buy from a reputable 3rd party, you will have to do the validation from scratch.

A note of caution: I started with a commercial certificate from StartSSL. They had the cheapest commercial plan for my purposes. Unfortunately, StartSSL was bought by WoSign, another certificate provider, without revealing this - meaning they had two votes in the CAB forum where they should have only had one. Due to this and other irregularities, Firefox and Chrome will no longer trust their certificates starting in January 2017. I have since moved all my websites to Let's Encrypt! So, research the trustworthiness of your certificate seller.

Here are some popular SSL certificate sellers with their current (December, 2016) price for a single domain / year with domain validation only. This list is not exhaustive - it is only intended to give you an idea of the range and a place to start. Note also that you can usually get a cheaper cost/year by buying a three year certificate.

• DigiCert - $175
• GeoTrust - $149
• GlobalSign - $179
• GoDaddy - $50.99
• InstantSSL - $79.95
• RapidSSL - $59
• SSLs.com - $8.95

Installing Certificates for HTTPS

How you install a certificate will, once again, depend on which panel / platform you have. MTA specific installation will be covered in the next part of the series.

• Plesk (https/mail)
• cPanel (https/mail)
• Apache (https)
• Windows/IIS (https)

Installing Certificates in Plesk

If you're using the Let's Encrypt! extension, congratulations! You're done for now.

Otherwise, as per the instructions:

• go to the "Hosting Settings" of the domain
• enable "SSL Support"
• go to the "SSL Certificates" of the domain
• select "Add SSL Certificate" (I've never gotten the "Upload certificate button here to work correctly)
• enter a name you will remember for that certificate
• if you received your certificate in file form (the most common case), upload the three files you need. Otherwise copy and paste what you need, then click on the appropriate "Upload Certificate" button.

Installing Certificates in cPanel

Here is the cPanel guide to installing certificates. Unlike Plesk, cPanel will let you choose which domain / certificate to use for the MTA.

Installing Certificates in Apache

If you're running under Linux, you are almost certainly using Apache. If you're not using a panel, then this is the right place.
Here is Apache's documentation on setting up SSL/TLS.

Installing Certificates in Microsoft IIS

If you're running Windows through a panel, it should handle configuring your webserver for you. If you're not using a panel, and you're using IIS, not Apache, then:

Here is the guide for configuring certificates in IIS 7.0.

In Closing

I highly recommend using certificates and HTTPS for your websites, even if you don't technically need them or only need them for your mail server:

• Too many ISPs are doing deep-packet inspection and some even do ad injection. Encryption prevents that.
• HTTPS, though theoretically slower than HTTP, is actually faster in practice.

I also highly recommend getting your certificates from Let's Encrypt! and, if you can, supporting them in their good work.

Thank you for reading my blog post!

Copyright © 2016 William David Herndon. All rights reserved.

This series:

• Part 1: The Pieces of the Puzzle
• Part 2: Subdomains and DNS
• Part 3: Encryption and SSL Certificates <- You are here
• Part 4: The Email Server(s)
• Part 5: Autoconfiguration
• Part 6: Debugging Email Servers

Softwareschmiede Herndon

Developer Stuff

Dev On Wheels

Religion

Baha'i Faith

Interfaith Explorer

Runden Tisch der Religionen / Marburg

Martial Arts

Taido Ryu Jujutsu

Dojo Wehrda

Guitar

Johannes Treml

Silvestre (Silver) Borrás Navaro

Paul Bowman (facebook)

Jody Cooper (facebook)

Anton Skrobanek (youtube)

Writing

Virtual Writing Group (forum)

Writing Excuses (podcast)

National Novel Writing Month (NaNoWriMo)

Other Stuff

Rukens Galerie

This series:

• Part 1: The Pieces of the Puzzle
• Part 2: Subdomains and DNS <- You are here
• Part 3: Encryption and SSL Certificates
• Part 4: The Email Server(s)
• Part 5: Autoconfiguration
• Part 6: Debugging Email Servers

Sub-Domains

Given a main domain named example.com, here are two configurations I will not be handling:

• Your ISP handles the mail: example.com DNS entries point to the ISP's mail handlers. See your ISP's documentation.
• You handle someone else's mail: their DNS entries and configuration files point to your servers. Do mail for yourself first, then worry about getting it right for third parties.

Here are some common configurations I will be covering:

• All in one: everything is handled on example.com. If you setup all your DNS records correctly and support autoconfiguration, then this should work fine.
• A single separate mail subdomain, usually mail.example.com. Sometimes with a separate autoconfig subdomain, but usually not.
• Separate servers for each protocol: pop.example.com, imap.example.com, smtp.example.com, plus usually separate autoconfig servers autoconfig.example.com, and/or autodiscover.example.com.

Some email clients will automatically check for the specific subdomains, either before or after checking DNS records for the subdomains. If your ability to make DNS records is limited in some ways and/or if your subdomains and their appropriate certificates are cheap, then you may want to go with option 2 or 3 above. Most small-operations will probably want to go with option 1, in which case you can skip straight to your DNS records.

Otherwise:

• Creating a Subdomain in Plesk
• Creating a Subodomain in cPanel
• Creating a Subdomain without a panel
• Creating a Subdomain under Windows

Creating a Subdomain in Plesk

If you are using Plesk, you should create your subdomains in their interface. it will take care of creating the DNS entry linking the subdomain to the IP-address and creating the necessary entries in Apache.

In Plesk, in the "Websites & Domains" section (the main section), there is a button near the top:

Press the "Add Subdomain" button, then fill in the subdomain and main domain. One can also change the sub-directory, and I usually do: the directory is usually created below the main domain anyway, so using the full domain name again: "softwareschmiede-herndon.de/mail.softwareschmiede-herndon.de" seems a bit of overkill to me. Then you click OK and it takes care of it.

You can then place the files you need below the directories you created for the respective subdomain(s).

Creating a Subdomain in cPanel

The documentation for creating a subdomain in cPanel is here. It is analogous to Plesk - automatically creating the necessary DNS records, changing the Apache config file and creating the base directory of the new subdomain.

You then place the files you need below the directories you created for the respective subdomain(s).

Creating a Subdomain with no panel (with Apache)

If you're not using a panel to control your subdomains, you may still want to "roll your own". You will need to make the entries in the DNS linking the subdomain to your IP-address, which you may want to do first.

Then you will need to add <VirtualHost> sections to your Apache configuration file. On my current version of Ubuntu that is "/etc/apache2/apache2.conf". You can find examples on the Apache website here, documentation here.

Creating a domain under Windows

In order to create a domain under Windows without a VPS panel, you will need Windows Server (2003, 2008, 2012) and Active Directory. When installing Active Directory you will also want to install the DNS Manager. Then you can start "dcpromo" from the command line or from the "run" command under Start. If no parameters are given, it will start a Wizard that will guide you through the process. It is also possible to do everything from the command line.

Once you have created the (sub-)domain, you still have to let IIS "know" about the domain: open the Internet Information Services (IIS) Manager, found in the "Administrative Tools" section of the Control Panel.

Open the tree below the server that should receive the subdomain (on the left of the screen). Right-click on the subsection "Sites". From the menu choose "Add Website...". Fill out the fields as needed and confirm.

Your DNS Records.

DNS is the Domain Name System. It is basically a distributed "telephone book", linking names to IP-Addresses and services. Your server is responsible for its own records, which get propagated to a higher level server (usually at your provider) and thence to the rest of the internet.

If you are creating subdomains or changing other DNS records, please remember that it may take up to several days for DNS records to propagate so that others can see the new subdomains or other changes.

Each DNS record has a short (1-3) letter Record Type that specifies what kind of info is in the record.

Basic DNS Records

Below are the basic DNS records on pretty much all servers. If you are creating subdomain without a panel, you will need to create an A record for each subdomain, the other basic records should already be correct, but are included for completeness.

A Record - Address Record

The A Record is the most essential record: it maps a (sub)domain name to an IP-address. Here are some entries from my own DNS records:

Note the trailing dot: that is not a mistake. That is actually part of a fully-qualified domain name. Most panels will automatically take care of the dot for you.

If you are creating subdomains without one of the panels that takes care of DNS for you, then you will need to make an A record for each of your subdomains. And don't forget the dot at the end.

German Shorthair Pointer

PTR Record - Pointer Record

The PTR Record is a reverse lookup. Someone wants to know a domain name from the IP-Address. There's only one of these per Server. If you're sharing a server, you won't be able to change this. The PTR record from my server:

NS Record - Name Server Record

The NS Record delegates your server's DNS-Zone to the specified name servers. This is DNS Bookkeeping: don't touch these unless you know what you're doing. These are my records, which probably won't mean much to you, but are essential for DNS lookups of my domains to work correctly.

CNAME Record - Canonical Name Record

The CNAME Record specifies an alias. It is optional, but most servers have one or more. If a CNAME record is found while looking up a name, then a retry will be done with the new name. An example from my server, aliasing "www." to the main domain:

TXT Record - Text Record

A Text Record is a generic record that can be and is used for many different things. One of the common uses is for site verification - proving to some entity that you have control of the site:

Another common use is for an SPF Record, which is relevant for eMails - see the TXT/SPF section below.

DNS Records relevant to eMail

You will want to create the following DNS records for your eMail server:

MX Record - Mail Exchange Record

The MX Record specifies the name of the server running the Mail Transfer Agent(s) (MTAs). You will want one of these records, even if you have no separate mail server: many eMail clients will get this wrong if there is no entry.

For large operations there can be multiple mail servers for each domain and hence multiple MX records, but you will only want one per domain. The number in parentheses is a priority number, which is ignored when there is only one.

Note that I do not have a separate mail server. If I did, the mail server name would be on the right under Value.

SPF Record - Sender Policy Framework Record

Support for the SPF Record was discontinued, but they often still exist. The TXT/SPF Record (below) is still used.

TXT/SPF Record - Text / Sender Policy Framework Record

SPF is an eMail validation system used to catch spoofers by verifying which servers are authorized to send emails from your domain.

It's complicated. Your panel probably takes care of this for you. If it doesn't, you'll want to duplicate the record below, with your domain substituted on the left. My TXT/SPF Record:

SRV Record - Service Locator Record

These records have an unusual format and are used to locate specific services like SMTP and IMAP. The format is:

_(service)._(protocol).(domain) <TTL> <class> SRV <priority> <weight> <port> <target>

Note that the underscore is not a valid character for a server URL, so the names can only be used for locating services.

My SRV Records, specifying my SMTP server and IMAP server respectively:

Now that you know which records you need, we'll cover how to view and edit your DNS Records. I cover four configurations:

• Plesk
• cPanel
• Ubuntu
• Windows IIS

DNS in Plesk

Under Plesk, go to "Websites & Domains" (home), scroll to the domain for which you wish to change the DNS records, then click on "DNS Settings" (circled in red in the image).

There you can add new records by clicking on "Add Record" or edit old records by clicking on the "Host" part of the record you want to change, like the IMAP SRV record circled in the image.

After you have made changes, an "Update" button will appear at the top. When you are done making changes, click on the "Update" button, and when that is done, click on the "Apply DNS Template" button.

Full documentation for DNS under Plesk is here.

DNS in cPanel

Documentation for changing DNS records in cPanel is here.

DNS in Ubuntu (no panel)

Under Ubuntu the DNS records are handled by a program called BIND (or BIND9). Full documentation can be found here. Below is a quick-start guide:

There should be a file "/etc/named.conf". If it isn't there, use "whereis named" to find it. Use "ls -l" to find where it is linked (if at all):

me@myserver.com:~$ ls -l /etc/named.conf
lrwxrwxrwx 1 root root 34 Dec  6 10:36 /etc/named.conf -> /var/named/run-root/etc/named.conf

Look at the contents of named.conf: if it says something like "THE FOLLOWING LINES WERE GENERATED BY PLESK", then do not edit it, but rather use the program which generated it to make changes: anything you do in such files will likely be overwritten.

The beginning of the file will be sections for options, key, and controls. In the options section there should be an entry for directory:

options {
listen-on-v6 { any; };
    allow-recursion {
            localhost;
    };
    version "none";
    directory "/var"; // <-----------------
    auth-nxdomain no;
    pid-file "/var/run/named/named.pid";
};

If named.conf was not linked, then that is the zone-file directory. If named.conf was linked, as in the example, then BIND will be running in a "chroot jail", and you must make that path relative to the chroot. In the example, "/var/named/run-root/var" is the zone-file directory.

The rest of named.conf will probably be the zones, which should look something like this:

zone "softwareschmiede-herndon.de" {
    type master;
    file "softwareschmiede-herndon.de";
    allow-transfer {
        5.35.246.86;
        80.237.128.10;
        common-allow-transfer;
    };
};

The two IP-Addresses are the Name Servers in the NS records. There is some info about the relationship between the servers, and the rest of the information is found in the file specified, in this example "softwareschmiede-herndon.de", which is found in the zone-file directory.

The zone-file (softwareschmiede-herndon.de), should look something like this:

$TTL    86400

@       IN      SOA     lvps5-35-246-86.dedicated.hosteurope.de. wdh.softwareschmiede-herndon.de. (
                    1481981362      ; Serial
                    10800   ; Refresh
                    3600    ; Retry
                    604800  ; Expire
                    10800 ) ; Minimum

softwareschmiede-herndon.de.             IN NS   lvps5-35-246-86.dedicated.hosteurope.de.
softwareschmiede-herndon.de.             IN NS   ns2.hans.hosteurope.de.
ipv4.softwareschmiede-herndon.de.                IN A    5.35.246.86
softwareschmiede-herndon.de.             IN A    5.35.246.86
blog.softwareschmiede-herndon.de.                IN A    5.35.246.86
ftp.softwareschmiede-herndon.de.                 IN CNAME        softwareschmiede-herndon.de.
www.softwareschmiede-herndon.de.                 IN CNAME        softwareschmiede-herndon.de.
softwareschmiede-herndon.de.             IN MX  10 softwareschmiede-herndon.de.
softwareschmiede-herndon.de.             IN TXT  "v=spf1 +a +mx -all"
_imap._tcp.softwareschmiede-herndon.de.          IN SRV 0 10 143 softwareschmiede-herndon.de.
_submission._tcp.softwareschmiede-herndon.de.            IN SRV 0 10 465 softwareschmiede-herndon.de.

The format of the different records is evident in the example. When done making changes, increase the number marked "; Serial" above - if it is not increased, the changes will not be propagated. Some recommend using the date to generate the number, "YYYYMMDD##", but it doesn't really matter as long as it increases.

DNS under Microsoft Windows

Managing DNS under Windows without a VPS panel is too complicated to cover here. Here is an overview of DNS for Windows Server 2008.

In Closing

If you can't change your DNS records or have only limited ability to do so, don't sweat it: most client software does not access DNS. Even Thunderbird, when it is looking up an MX record, does not look it up directly, but rather uses HTTPS to ask a service to do the look up for it.

Thank you for reading my blog post!

Copyright © 2016 William David Herndon. All rights reserved.

This series:

• Part 1: The Pieces of the Puzzle
• Part 2: Subdomains and DNS <- You are here
• Part 3: Encryption and SSL Certificates
• Part 4: The Email Server(s)
• Part 5: Autoconfiguration
• Part 6: Debugging Email Servers

This was supposed to be a single blog post. But it has expanded until there is no way to do it in a single, reasonable sized post. This post is an introduction to the resources you will need to know for the rest of the series. You can skip the parts of the introduction you're familiar with, without missing anything important.

This series:

• Part 1: The Pieces of the Puzzle <- You are here
• Part 2: Subdomains and DNS
• Part 3: Encryption and SSL Certificates
• Part 4: The Email Server(s)
• Part 5: Autoconfiguration
• Part 6: Debugging Email Servers

The Goal

You have your own website or websites, and you want to provide your users with email. You want your user to enter their email address and their password into their favorite email-client application, and it should just work.

It should also be secure, so that it will continue to "just work" for your customers.

If you are providing the service yourself, you may or may not want to offer your user a choice of account types:

If you're using an ISP that provides email, and it just works, congratulations! You can skip out on this blog post.
Otherwise your goal is that the above should be the end of user setup. Anything else and your users will be calling you for support.

The Problem

So, the user chooses their account type, and instead of it just working, the user is presented with screens like these:

If they get anything wrong, they'll get an error message they can't interpret (and perhaps you can't either).

Or worse. They may get a scary screen like this.

The Pieces of the Puzzle

So, there are a lot of pieces to this puzzle. I have short sections on each linked below.

• The account types / Email-protocols: POP, IMAP, SMTP, Exchange / WebDAV
• The email server programs (daemons): sendmail, Postfix, Exim and Microsoft Exchange Server cover 90% of the market.
• The email clients: iPhones, Microsoft Outlook, Mozilla Thunderbird, WebMail and Android mobile phone email clients.
• DNS - Domain Name System - the system by which clients find your server(s) and some information about it/them.
• Autoconfiguration - different for different clients
• SSL/TLS and Certificates - so connections are secure and your client knows that it is really connecting to the correct server.

Which pieces you will want to use depends on your and your user's requirements and limitations. Getting all these pieces to play nice together is a difficult dance, but I'm going to try and walk you through it here.

The Account Type / Email-Protocols

Incoming Protocol: POP - Post Office Protocol

Here the name is the program: the server acts like a post office, and when your client picks up your email, it is deleted from the server. This mode is sometimes called "download-and-delete" or "maildrop". Often called POP3, because version 3 is the last major version, the current formal documentation is RFC 1939).
Historically, this was the only way to do things: you had multiple servers between sender and receiver, and servers were often offline, so each server on the way kept your email until it was picked up by the next station. Now servers usually talk directly to each other.
This may still be a good protocol choice if you want to minimize the amount of space email takes up on the server, or if your users don't want copies of their email on the server. On the other hand, if your users want to access their email from their mobile, their tablet and their desktop without missing a beat, then they'll need the emails to stay on your server where they can access them. That's where the major alternative, IMAP, comes in.
A modern POP3 server listens on port 110 and may thence switch to encrypted protocol, or it may listen for encrypted traffic directly (POP3S) on port 995, or both.

Incoming Protocol: IMAP - Internet Message Access Protocol

This protocol allows a client to manipulate messages (move between folders, copy, delete) and retrieve messages stored on a server. Messages are only deleted on the server when explicitly commanded to do so, and the client normally only keeps a small cache of messages. The current version of this protocol, version 4, was formalized in 2003 with RFC 3501.
Depending on space requirements, you may need to agree with your users on an archiving scheme, so that emails that they need access to remain online, while older emails are stored in an offline archive and deleted from the server.
A modern IMAP server listens on port 143 and may thence switch to encrypted protocol, or it may listen for encrypted traffic directly (IMAPS) on port 993, or both.

Outgoing Protocol: SMTP - Simple Mail Transfer Protocol

This protocol is for sending messages and is used by all major mail servers. The current version was formally defined in RFC 5321.
Although SMTP servers listen on port 25, this is usually reserved for unencrypted traffic between servers. They listen for user email-clients using encrypted traffic directly (SMTPS) on port 587 or port 465 (deprecated), or both.

Microsoft Exchange Server / WebDAV - Web Distributed Authoring and Versioning

This proprietary system from Microsoft works like the IMAP/SMTP combination: messages are kept on the server and it allows you to send messages. WebDAV is a generic method of manipulating data via HTTP, that Microsoft extended for manipulating mail messages on Exchange Server. WebDAV support was discontinued in Exchange Server 2010, replaced by Exchange Web Services (EWS), a SOAP/XML based API.
Since Microsoft Exchange Server supports POP3, IMAP and SMTP anyway, I'm going to leave out this proprietary nightmare that would expand this already too large article.

The Email Server

The big four Mail Server Softwares, often called Message Transfer Agents (MTA), are Exim (ca. 50% market share), Postfix (ca. 30%), sendmail (ca. 8%), and Microsoft Exchange Server (ca. 3%). All except the Exchange Server are Open Source. Depending on your setup, you may not have any choice in which MTA you use. Those that have a choice are usually using a virtual private server (VPS), or a dedicated server.
If you have a choice, any of the four will do - they all support POP3, IMAP and SMTP.

sendmail

The oldest of the three, this Open Source software was first released in 1983. It has great flexibility, but configuring it is daunting.

Postfix

Originally released in 1998 as a replacement for sendmail, Postfix is still under active development. It has integrated protection against spambots and malware. It is released under the IBM Public License.

Exim

The default email program on Debian/Linux distributions, it has over 50% market share. It is often used together with cPanel, a major web-hosting control panel.

Microsoft Exchange Server

Microsoft Exchange Server is proprietary software. In addition to the standard protocols, it supports its own protocol, as mentioned above. It runs only on Windows Operating System, and supports a calendar and other functions as well as email.

The Email Clients

The Big Clients

You probably want to support these:

• Apple iPhone/iPad - the largest by market share, no choice for those that own one
• GMail - not just for Google Email accounts
• Microsoft Outlook - still many people using Windows 7/8 desktop
• Mozilla Thunderbird - for Linux users and picky Windows users
• Windows 10 Mail - comes with Windows 10

WebMail

You probably also want to let your users access email via browser with a WebMail program: e.g. Horde, SquirrelMail, RoundCube.

Smaller Clients

You may or may not want to support these smaller clients (I use K-9 Mail myself):

• Android clients: K-9 Mail, MailDroid, Blue Mail, et al.
• Desktop clients: Claws Mail, TheBat, et al.

No Worries

There are some popular email clients that have restrictions preventing their use with the mail servers we're talking about here. That is not always obvious: e.g. I installed Touchdown to test it, before discovering I couldn't use it. You won't need to worry about:

• Touchdown - only supports Microsoft Exchange Server
• Boomerang - only GMail accounts
• Yahoo! Mail - only Yahoo accounts
• AOL Mail - only AOL accounts

DNS - Domain Name System

The Domain Name System has different types of records. In rough order of frequency of usage:

• A - maps names to IP-Adresses
• NS - for authoritative name servers
• CNAME - for aliases
• TXT - miscellaneous text records:
  • Verification records: e.g. Google, Office 365
  • SPF - Sender Policy Framework - should be TXT duplicate of SPF record.
• MX - the "Mail Transfer" record, giving the hostname of the mail server
• SRV - The service record, telling the hostname and port number of specified services
• SPF - Sender Policy Framework gives info about your email policy to others. This may be a duplicate of an SPF record.

Linux has several commands for looking up DNS records. The most common is host. Under Windows you use NSLookup.

In either case, you should use a lookup program to verify that your record changes are correct and are being propagated. Best is if you can check the records on your server itself first: propagation to other servers may take hours or even days.

Autoconfiguration

Several Email client writers have tried to mitigate the difficulties of configuring their clients by documenting methods of specifying to their client how they should be configured.

Outlook Autoconfiguration

One can tell Outlook how to configure emails by placing an XML-file with the necessary information in a specific location on the main domain, or on an "autodiscover" subdomain. One can also place a redirect on the "autodiscover" subdomain to a secure server or make a SRV-record in DNS. The XML file can be static, or it can be generated dynamically at each request - the file request is done via POST with the username passed. The details are here.

Mozilla Thunderbird Autoconfiguration

Similar to Outlook, one can place an XML file in a specific location, but the location is different: either on the server specified in the DNS MX-record or in the "autoconfig" subdomain. Only a static file is expected, but it supports some variable names, like "%EMAILADDRESS%". The details are here.

SSL/TLS and Certificates

The original email did not have any security, but now it is an absolute requirement: with so many spammers, email servers must properly authenticate everything over a secure connection.

Depending on how you configure your server(s), you may have many subdomains, e.g.: pop3.mydomain.com, imap.mydomain.com, smtp.mydomain.com, webmail.mydomain.com, autodiscover.mydomain.com, autoconfig.mydomain.com. If so, you will either need a wildcard certificate or separate certificates for each subdomain.

Copyright © 2016 William David Herndon. All rights reserved.

This series:

• Part 1: The Pieces of the Puzzle <- You are here
• Part 2: Subdomains and DNS
• Part 3: Encryption and SSL Certificates
• Part 4: The Email Server(s)
• Part 5: Autoconfiguration
• Part 6: Debugging Email Servers

It's been a while. I've been busy and I set too high standards for my posts to keep up a weekly pace. So, I'm going to try and make shorter simpler posts more frequently.

Here are the highlights of what I've been busy with:

Installing Android Studio

I installed Android Studio on a virtual machine as a start on my new series on Android development. It was very different than the first time I did it, and it took way more space than I was expecting. And then I got interrupted...

Dealing with SSL/TLS Certificates

I have fallout from the impending removal of StartCom/StartSSL's certificate from Firefox's and Chrome's list of trusted certificates. Here is a Mozilla blogpost about it.

They were the cheapest commercial source for trusted certificates, and I got certificates for my own and some websites I maintain, for a little over 100€ / year total. The next best commercial alternative would have cost me thousands of Euros for the same thing and been less flexible.

Basically, I agree with the reasons for decertification, but that left me (and thousands of others) with no viable commercial source. It disgusts me, that commercial sources require large amounts of money for little to no work. After verifying that you are who you say you are, it's basically a click of a button to create a certificate. Yet all the certificate providers (except StartCOM) charge large amounts of money for a certificate, and extra for every sub-domain of a domain you've already shown you own.

Enter non-commercial Let's Encrypt! The certificates they issue last only three months, but they're completely free. I found a plugin for Plesk (the software I use to control my VPS) that made the transition relatively painless. Once set up, it is a lot easier to install a "Let's Encrypt" certificate than a regular certificate. I only had a few issues with websites using ProxyPass, but that's all fixed now.

The issue of SSL/TLS is pretty important, I'll probably do a whole blog post on it later.

A Virtual Writing Group

There were several reasons for this. A primary motivator is that I need experience with phpBB for a paying project I'm working on. The requirement is for writing serialized documentation, where different people can write entries, but may not change other's entries, and when they change their own entries, those changes are documented. This is basically the same as threads in a bulletin board with login and edit logging, so that's what I will be installing for the project.

I've also been a hobby-writer for a long time. Years ago I was a member of a similar website, RateYourWriting.com, that is now defunct. I want a little of that flavor and a bit of the regular "writing group" flavor. I don't think a regular English-language writing group is really possible here in Marburg. Aside from the setting up phpBB I've learned about phpBB extensions and programming in PHP - the edit logging is also useful for the writing group and I created a word count extension. I will be releasing it to Open Source once I've got internationalization working correctly.

A blog for my father on Multiple Sclerosis

I created a blog for my father and have spent a bit of time helping him with it. WordPress is soooo much easier than ghost.

Writing a WebApp from scratch

There's been a feeler for a project that would start next year, and would involve writing a WebApp. I'm pretty sure I can sell this WebApp to others as well, so I'm getting a head start on it. And if it doesn't work out, at least I'll have gotten some more practice in PHP.

Installing Ubuntu Linux

I installed Ubuntu on an old laptop that had XP, that didn't have enough space for the latest Windows XP update. The laptop also has a problem with the DVD drive - there's a loose connection, but if I hold the drive in a particular way, it will work. So I spent a few hours holding a DVD drive so it would boot and install from the boot DVD. I'm a bit surprised that Ubuntu no longer supports boot/install CDs - supporting older, resource starved devices used to be one of the big reasons for using Linux.

Bluetooth

First I wanted to listen to podcasts on my car radio. My car radio supports bluetooth and my smart phone supports bluetooth. After some fiddling, I got it to work.

Then I got a cheap bluetooth headset. Getting that to work with my smart phone was a little bit fiddly, too, but not too much trouble.

Trying to get bluetooth to work properly on my Windows 7 64-Bit was an utter failure. Apparently the standard bluetooth driver for my bluetooth dongle under Windows 64-bit is missing support for a lot of devices, so they recommend installing Windows 7 32-Bit. Grrrr. I may revisit this.

Happy Thanksgiving! And boycott Black Friday! Spend the time with family and friends.

Copyright © 2016 William David Herndon. All rights reserved.

In programming, as in most professions, the tools make the master of the craft. You have to know which tool to use when, and you have to be able to use them well.

My goals in this article are: (1) to act as an introduction to the tools of the trade for those looking to try programming (2) to give some tips to those who are already programming and (3) as a reference for future posts - you should be seeing a lot of these tools in my future posts.

Since this entire post is about resources (tools are resources), there will be no "resources" at the bottom of the post; instead there will be an index to quickly find the section you want.

Without further ado, I humbly present what I consider to be the most important tools for the software smith:

Your Own Brain

You really need to treat your brain like the tool it is: you have to keep it sharp and treat yourself right:

• Take frequent short breaks: 10-15 min per hour is a good rule of thumb.¹
• Make interruption-free time for complex tasks.²
• Get enough sleep (I'm afraid I often fail at this one). The problem for me is, the best interruption-free time is late at night...
• Eat well - your brain consumes a lot of energy and the supply needs to be consistent. If you're eating lots of sugar, your blood sugar will be bouncing like a ball and your cognitive abilities will, too.
• Exercise - this improves circulation and gives you a nice shot of dopamines, which helps you concentrate better.
• Learn new, interesting stuff - "exercising" your brain helps keep it sharp through less interesting tasks.

A Search Engine / Good "Google-Fu"

Most people don't think of this as a programming tool, but, after the brain, it is the most important tool we have these days. When I was learning to program, search engines did not exist. Nowadays, a good search engine is completely indispensable. Here are a few basic scenarios where I need it:

There should be a picture of me applying GoogleFu to a problem here.

• Something has broken, and I need more information on how to fix it: has anyone had this problem before? If they solved it, how did they do that? Did they discover a work-around? What does this obscure error message really mean? Before search engines, you would be writing emails and waiting days for responses, or if it was really urgent, paying for expensive long-distance phone calls.
• I need the name and/or parameters of some function in some language or library I'm less familiar with. Way back when, you could keep all the references on your shelf. Now, with new languages and platforms coming out several times a year, you would need a whole house just for the books, and a budget to match.
• I'm trying to learn some new language or platform, and the guide I'm reading starts referencing technology x and paradigm y that I've never heard of before.

The simplest search sometimes work, but as often as not, you'll need to think of 10 different ways to search for the same thing before you hit gold. Good Google-Fu is important.

Google is still the best, in my opinion, but has actually been going down in utility for me as they've been tuning it more towards my preferences, e.g. "no, I really don't want the Java version of this function, that's why I typed "Python". In quotes. And you're still ignoring it and giving me pages where the word 'Python' isn't present!"

For most of my daily needs, I use DuckDuckGo: I'm not entirely comfortable with the way Google sometimes seems to know more about me than I do. When I'm really not finding anything at DuckDuckGo, I go back to Google.

Touch Typing

Technically this is more a skill than a tool, but I'm still including it: it's really important.

When I was 14, I decided to learn to touch type. It took about a month, filling at least two pages every day. I was lucky: I was at grandma's and got to practice on her electric typewriter, rather than on my mother's manual. In the 40 years since then, it has paid me back at least 1000-fold. That is not hyperbole, it is understatement.

In programming, more than in most other professions, you need to be "in the zone" to work well. To me, having to look down from the screen to hunt-and-peck just doesn't bear thinking about.

If you don't have one, consider getting yourself a decent keyboard. A laptop keyboard may do in a pinch, but really, you need something better. My main development computer is a laptop, but I have a separate keyboard for it.

A (Powerful) Text Editor

I could fill a long post on text editors alone: this section was taking over half of this post and expanding, before I decided to cut it drastically.

It is fairly likely that at some point you're going to be bopping around in remote shells, installing stuff and fixing problems. When you do, you will need an editor that:

• can do everything from the keyboard.
• supports advanced functions like keyboard macros
• supports multiple windows and buffers, with copy/paste between them.

There are two major text editors that fit this criteria: Emacs and Vim.

The original Emacs was written in 1976 by Richard M. Stallman (yes, that guy) and Guy L. Steele, Jr. (I learned Lisp from Guy at C-MU).

Vim is an Open Source clone of vi, a text editor written to be part of the POSIX specification. Vi was also created in the late 70s.

You should know one of them well enough that you can do buffer-switching and other advanced commands without having to pause to look it up. You should, at a minimum, know how to exit your less preferred editor.

These days, if you know neither, you should probably give the preference to Vim, because some flavor of Vim or vi is on nearly all Linux installations (due to POSIX spec), while this is not true of Emacs. Emacs is my own preferred editor.

Operating System, Shell, and Shell-Scripting

Over the years I've run into a fair few people who were of the opinion that shells are passé, that all you need is the GUI of your operating system and a good IDE, and you never need to touch those clunky old shell's with their impossible to remember commands again.

If you want to be an average programmer and never touch the bleeding edge again, I'd agree with that. Otherwise, Exhibit A: setting up this blog would not have been possible without extensive use of a Linux shell (bash). When the init-script for ghost didn't work for my system, I rolled my own. Not a serious problem when you understand the scripting language and have decent Google-Fu.

"But I work under Windows!" So do I. Windows 7 is still the primary OS on my development computer. I work with a shell open at all times. I have several dozen cmd shell scripts for Windows that I wrote myself to make my life easier. Maybe I should upload the most useful of them and make a post about it. If you would like that, comment below and let me know.

As a developer, your most common choices for operating systems are: Windows, Linux, and MacOS (OSX). I will not attempt to get into the OS wars here. Suffice to say, your choice will be heavily influenced by your target audience and your budget.

Miscellaneous: there are multiple *nix shells, bash being the most common. It can sometimes be a problem getting a script to work in all shells being used. MacOS is based on Unix, so if you're familiar with Linux' bash, the MacOS shell isn't too hard.

A Secure Remote Shell

If you're doing anything with VPS, or even just with a server in your local network, then the need for remote access is pretty obvious. For *nix to *nix you'll want to use OpenSSH - the ssh command and sshd daemon. For Windows client to Linux server, I use putty. For Windows to Windows, Remote Desktop Connection is my choice. I've never really needed Linux to Windows, but if I do, I'll probably just use OpenSSH for client and server, with the sshd on the Windows side running under CygWin.

Browsers

Browsers have morphed into platforms in their own right. The following are the top browsers in rough order of market share:

Name	Logo	Platform(s)	Notes
Chrome		on all popular platforms
Firefox		on all popular platforms
Internet Explorer		Microsoft Windows only.
Safari		iOS/macOS only.
Opera		on all popular platforms

Currently, I try to make sure that all my HTML/CSS/DOM is correctly interpreted by Chrome, Firefox and IE, which covers over 90% of my target audience. When I switch to Linux as my primary platform, I will likely drop IE in favor of Opera.

Here are some other browsers which may be of interest:

• Microsoft Edge- web browser that comes with Windows 10. Windows only. Interestingly, as of this writing, there are a number of non-Microsoft browsers in the Play Store that have the word "edge" in their title.
• Lynx - text-only browser.

Email Client

As a programmer, you need email, but you probably don't need much by way of extras, so this section is short. Most OSes come with an email client, and many mail providers have a webmail interface. They are generally not complicated.

The company I worked for had me using Outlook³ with Microsoft Exchange Server for many years, while I used the webmail interface of AOL and a more local mail service for private email. That was acceptable.

Now I've switched to accessing all of my email addresses using Thunderbird. I like this better.

(I have both my own domain and an AOL address. What does that say about me?)

(Secure) File Copy

My primary tools for transferring files are:

• FileZilla - A graphical "ftp solution". I use this mostly to update websites.
• scp - Secure CoPy, a basic *nix tool. Under Windows I will sometimes open CygWin shell just to use scp to copy a file to/from a folder not reachable by ftp. I should probably look into a native SCP for Windows - I use it often enough.
• curl - a data transfer program, supporting ftp(s), http(s), and an insane number of other formats.
• Default ftp client - both Linux and Windows have a default command line ftp client.

Version Control

I've been using some type of version control since ca. 1990, long before it became de rigueur - it has saved my butt innumerable times. Even when it wasn't saving my butt, it allowed me to proceed more quickly when making far reaching changes: a simple click or command for backup and after a short pause, I can proceed.

Under Visual Studio I originally used Visual SourceSafe (VSS). There were numerous problems with it, especially through remote access (Darn it, Microsoft! Who doesn't use version control remotely?!). But it was still better than no version control. Later I used Team Foundation with Visual Studio. It did its job well.

For side projects I used to use Concurrent Versions System (cvs), then Subversion (svn). I was happy with both.

Now I use git for everything, it works well on the command line and is integrated into most of the IDEs I use.

File Difference

I find it necessary to look at the difference between files in three cases: (1) I am about to commit my changes, and I want to check that the changes are clean (no debug output, comments cleaned up, etc.), (2) to merge conflicting versions and (3) to review someone else's changes.

Under Windows I still use windiff, which came with the Microsoft Windows SDK, but is no longer supported. I'm going to miss it when it stops working.

Under Linux or when I'm restricted to text-only, I use the standard Linux utility diff.

Text Search

Under DOS and then later Windows I used the Norton Utilities program ts for text-search. After moving to 32-bit, I actually wrote my own replacement - I found it overblown to pay for a program I could write inside of an hour.

Under Linux I use grep, of course

Virtualization / Emulation Software

As a developer for commercial programs, the programs usually need(ed) to run on multiple versions of Windows (or other OS). Way back when, we had to have separate machines for each operating system. Now I use Innotek Sun Oracle VirtualBox. I have Windows in four flavors, Linux in two flavors and even Android (technically also Linux) in three flavors, all on my server. I don't use the server itself for anything else.

When I need a Unix-like environment under Windows, where you can use *nix tools and scripts, I use CygWin.

Integrated Development Environment (IDE)

IDEs generally try to be the be-all, end-all environment that you never have to leave, with integrated text-editing, compilation, debugging and other tools. The IDE of your choice usually depends on your developer platform and target platform.

• The most versatile Open Source IDE is Eclipse. You can configure it to target almost any platform, including Windows, but...

• If your target is Microsoft Windows, then you'll likely be using Microsoft Visual Studio. It's actually pretty decent for other targets, too.

• If you're programming for Android, you'll most likely be using Android Studio

• Emacs and Vim - yes, both of these text editors can be used as an IDE. If you're in a text-only remote shell, this can be important.

• GDB - not really, but: if while debugging you type Ctrl-X "A", it will enter "TUI Mode", which will allow windowed debugging as if in an IDE, with Emacs-like commands.

I've used all of the above, and am actually pretty happy with all of them.

Compiler/Interpreter

Not going to say much here. There's too many, and the choice is often made for you, depending on what you want to do. I used to make short programs, like calculating the prime numbers under a thousand, in gwbasic, but that has stopped working. Now I usually do things like that in C/C++ using gcc.

Index

• The Brain
• Search Engine
• Touch Typing
• Powerful Text Editor
• OS / Shell Scripting
• Secure Remote Shell
• Browsers
• Email Client
• File Copy
• Version Control
• File Difference
• Text Search
• Virtualization Software
• Integrated Development Environment
• Compiler/Interpreter

Footnotes

1 Article from The Atlantic, 2014 "A Formula for Perfect Productivity: Work for 52 Minutes, Break for 17"
2 This is why you shouldn't interrupt a programmer - The Slightly Disgruntled Scientist
3 Outlook full version, not to be confused with Outlook Express.

Copyright © 2016 William David Herndon. All rights reserved.

Tell me a little about yourself, what you do and where you work.

So, my name is Matthias Labsch, I am a 34 year old front-end developer living in Freiburg, Germany. Currently I am working for a small marketing agency where I am responsible for front-end development using HTML 5, CSS 3 and AJAX on a WordPress basis. My main focus is Responsive Web Design and Accessibility.

Is there a special reason for the focus on accessibility? Your target customer?

During my work in several companies, it crossed my mind that most of the techniques I used for valid HTML pages and Responsive Web Design (RWD) are also a step closer to accessible websites. To name an example:

If you're using a more specific <input type="number" /> instead of a simple <input type="text" /> (which also accepts numbers), you will greatly enhance the possibility of screen readers to tell the user what kind of information they are requested to enter. As a byproduct, mobile devices will use a customized keyboard (e. g. only numbers) to assist the user.

My own handicap^* is just another reason to dive deeper into accessibility, as I am not dependent on special devices to use my computer or the internet. But while I am able to get my information from the internet very easily, I know many other people with disabilities who depend on assistive devices and techniques.

So, your target customer is not the reason - it's just plain good design to be accessible?

Exactly! My opinion is: if I can create an accessible website with a small effort, why shouldn't I? Most techniques for RWD are one step in this direction, so I am eager to go the next step as well. But there's still a lot to learn 😊

So, getting to the main theme: How would you define "Responsive Web Design"?

RWD is a collection of many techniques to enable a website to adapt to different viewports without changing its source code. The most important approaches are (in my humble opinion) Mobile First, Progressive Enhancement and a good content structure. On top of that developers have a wide range of techniques like fluid grids, responsive images and so on.

What are the basic principles of RWD?

Let's take the three approaches I mentioned before.

Mobile First is the concept where you develop your website and content so that mobile devices are favoured above desktop computers. This will force both the developer and the content engineer to emphasize the most important content first and create the source code accordingly.

This leads to the second approach, Progressive Enhancement. With Progressive Enhancement, you will enhance your content with more information and your website with more features, e. g. on your mobile device you will only see the websites header, the selected article and a footer. On a tablet, an additional sidebar will be visible and on a desktop computer you will see a fancy slider above the content. This way, your websites performance will be increased because you don't have to hide bandwidth-intensive features like the slider on mobile devices.

To achieve this, there are many good techniques and architectures. Many frameworks like Twitter Bootstrap or Foundation offer a wide range of features while architectures like ITCSS (Inverted Triangle architecture for CSS), OOCSS (Object Oriented CSS) or BEM (Block, Element, Modifier) will help you to organize your CSS.

"Graceful Degradation" is a decades old concept that applies to other engineering fields, too: when resources are limited, fall back to less resource intensive alternatives and/or leave things out. Is there a real practical difference between this and "Progressive Enhancement"? Or is it just a difference in where you start?

Both principles are legitimate and depend on your customer: if my customer wants to have a desktop-only website and decides later on that a mobile version is a must-have now, I would use Graceful Degradation - but with a huge cramp in my stomach. Experience shows at some point every customer wants to have a mobile version of their website. The reasons are manifold: bad Google ranking, user feedback, a change of mind and so on.

So for me, my preferred approach would be Progressive Enhancement. And as I said earlier: the code will be more performant. Just imagine that you can build up your CSS in a way that is cumulative, meaning the styles for the mobile viewport are loaded first, after that only changes for the next viewports are loaded because they enhance the code before. I do not want to condemn Graceful Degradation, but it's not my favoured approach.

So, you're saying there really is a practical difference: that (usually) large chunks of the website are loaded before being discarded in the graceful degradation approach?
Is that a fair paraphrasing?

Of course! Normally, you will have one CSS file (independent of whether you're using a CSS precompiler or not), so with Graceful Degradation you will find the source for e. g. the slider near the top of the file - which is loaded by a mobile device, but not executed. The mobile device now has to crawl its way down to the breakpoint responsible for its viewport and "forget" the loaded code.

Way better: load the code for the mobile viewport first and then ignore the following code because it is code for another viewport.

So, tell me a little about some of these terms you've mentioned. What about "responsive images"?

Regarding images, one major problem of RWD is: how to deal with images? It is possible to scale images, of course, but this is not the best solution in terms of performance.

One solution could be srcset, where you define multiple sources of an image within the srcset attribute in relation to a given viewport to display images of different quality in the viewport. A step further away is picturefill, where you are using the <picture> element to display different images within the changing viewports. This enhances the possibilities of srcset with the option of "art direction": you can use other images with their own srcset to e. g. display just one prominent tree (in mobile viewport) while you display the whole forest on desktop.

Regarding responsive images: if you want to have further information, this is a good article (German): "Responsive Images" at the blog "Kulturbanause". Google translation of same.

And "fluid grids"? Is that anything like CSS3 flexible box?

Here is a link to some examples: Fluid Grid examples from Twitter Bootstrap

It's the most beautiful way to distribute your columns, because you can say it doesn't make sense to display a 25% width column on a mobile viewport, so I will use my breakpoints to say that on a mobile viewport, every column is 100% width and my information is stacked on each other. Then I can distribute my columns in the way I want to. This is a slightly advanced usage of fluid grids, but mainly fluid grids means you have a percentage value for your columns and thus you're able to have this very fluid design as you can see here.

One of the major uses of columns is to get things next to each other horizontally. This kind of stacking one over the other doesn't allow the user to move horizontally in a mobile viewport as he would on a desktop. Like with a table, where you want to be able to look at adjacent elements both horizontally and vertically. How yould you do that?

There are several solutions. I cannot explain them in detail, because I have to look it up. But there's a project called Responsive Patterns library, and they have different solutions for that problem. One solution would be that you use colors so that you can tell which row you are in when scrolling. Or you can make use of "data tables", a JavaScript library which allows you to sort your table and filter your table and so forth. In some cases you might want to display your table as a pie chart or other chart in general.

<picture> is HTML5, and picturefill, if I'm rightly informed, is even more cutting edge. Do you run into problems with browser support?

<picture> is not supported by old IE browsers, picturefill is a JavaScript polyfill library to enhance these browsers with the new element. If you need further information which browsers are supporting the <picture> element, have a look at "Can I Use"

So if you're looking for "picture" you will see that IE Edge will support <picture>, but none of the older versions of IE. picturefill.js will enable <picture> support down to IE 8 as far as I know ...

It looks like people stuck with an old Android Browser are also out of luck.
Is that an issue for you or your customers?

Not really because most of our customers agreed on our recommendation to support the two latest versions of each browser, if the market share is significant.

A good source for responsive workarounds: Responsive Patterns

What is "Adaptive Web Design" and how is it different than "Responsive"?

Uh, that's a tricky question because most techniques used for responsive design are also used for adaptive design 😄 I would say that adaptive design just distinguishes between viewports and loads a design just for that specific viewport while responsive design tries to fit in the marginal differences between the devices. As an example: adaptive design will provide a 320 px wide design for all mobile devices while responsive design is able to cope with slight width differences between Android and iPhone versions.

Or simplified: adaptive design is a lazy version of responsive design because it just shows one design for a bunch of device viewports 😄

When should I not care about Responsive Web Design?

When it is clear for you and your customer that they will not have a mobile version of their website, e. g. when the website will run in an intranet exclusively.

Another reason could be the costs for a mobile version.

You were talking about accessibility earlier - do you support blind users who might be using a text-only browser, such as lynx?
Conversely, do you have any recommendation for a blind user?

To answer your first question: I would hide certain areas of my content with display: none and mark it with ARIA labels so that the screen reader will ignore it.

Unfortunately, I never used a screen reader for testing, so I cannot give any recommendations. But I think I will due to the blindness of my aunt and a friend of mine.

A lot of interactive services are now in both Web and App form, such as facebook and memrise. Do you have any comments on this trend?
(I suspect they do this, because they find RWD too hard)

The more I am using Facebook and YouTube on my smartphone, the more I like the responsive browser versions - simply because I can use an ad blocker there to get rid off these bloody adverts 😄

😊
Most web sites these days are based on some CMS framework: WordPress, Joomla, Drupal, et al.
Do some of them support Responsive Web Design better than others?
What should I look for in a CMS?

I am unfortunately not familiar with Drupal or Joomla, but the latest update of WordPress introduced built-in support for responsive images (via srcset).

Another important thing should be that the CMS is able to use themes (which can be responsive then) and that it should be possible to minimize and concat source files for CSS and JS. This is not directly related to RWD, but is a sideshow regarding performance (which is a must-have when we are talking about RWD).

Okay. Conversely, any advice for experienced programmers looking to do things themselves?
What further articles, books, or other resources do you recommend?
Any particular authors/bloggers?

Okay, if you want to start as an experienced developer you should get familiar with the basic principles of Mobile First, Progressive Enhancement and Content Choreography. Next, try to study common principles (fluid grid, responsive images, forms) by looking at a CSS framework like Twitter Bootstrap and then looking up the principles used there by reading the articles of some very awesome front-end evangelists, e. g. Brad Frost, Luke Wroblewski, Ethan Marcotte or Andy Clarke. While reading their articles, you will automatically stumble over more interesting topics which will help to improve your (responsive) web design: Atomic Design, Responsive Design Pattern and so on.

For general information and inspiration, please have a look at Smashing Magazine - it's the largest and most well-known magazine about web development. And they are based in Freiburg, too! 😊 And last but not least, ask me or have a look at my website, Dev On Wheels 😉

Thank you very much! 😊

You're welcome! That was fun! 😊

Lessons

• Interviews are fun!
• There's always more to learn.

Resources

• Twitter Bootstrap (home)
  Bootstrap Resources
  Fluid Grid Examples
• Zurb Foundation
• ITCSS (Inverted Triangle architecture for CSS) - Explanatory article at Creative Bloq
  itcss.io - which leads to the ITCSS twitter feed
  The ITCSS Primer article (at Net Magazine or elsewhere) appears to be missing.
• OOCSS (Object Oriented CSS)
• BEM (Block, Element, Modifier)
• Blog "Kulturbanause" (German language)
  Article "Responsive Images" / Google translation
• Responsive Patterns
• "Can I use" - enter a feature and see which browser versions support the feature and what percentage of users are covered.
• Brad Frost's blog
• Luke Wroblewski's website
• Ethan Marcotte's website
• Andy Clarke's Podcast, "Unfinished Business"
• Smashing Magazine
• Matthias Labsch's website "Dev On Wheels"

Footnotes

^*Matthias has a rare form of muscular dystrophy

Copyright © 2016 William David Herndon. All rights reserved.

With all the ads I've been seeing for and by Google Adsense everywhere, I thought it would be simple to sign up. Maybe not the big money-earner, but it should be simple, right?

Ha.

Finding Sense

Before this blog even existed, I thought I'd go ahead and register for it. The first problem was even getting to Adsense: I had seen ads for it, but when I went to my Google business account, it wasn't in the menu. Nowhere was there a link. Seriously? I need to google it? I didn't even remember the name - Google Ad-something. sigh

Google AdSense is here.

Does it make sense for your website?

One needs a Google business account with all that entails, but I already had that^*. I was already advertising my business with Google AdWords.

After creating an AdSense account (or whatever you call linking up your account with AdSense), you enter "Your website". I put in the website "http://www.softwareschmiede-herndon.de", which is pretty much the same now as it was then. (Which reminds me, I should add a link to this blog in latest news! done)

I got a message, telling me that the website wasn't really suitable for AdSense, that the details would be in an email. I didn't think to look more seriously at the message, it would all be in the email, right?

I never got an email.

I check my spam folder every day, too. Not just when I'm expecting an email.

I figured my website really wasn't very suitable - it's mainly an advertisement for my business, who's going to want to see someone else's ads there as well? Plus it didn't have a privacy policy yet - it didn't use cookies, there was no registration, why would it need a privacy policy? So, I decided to wait until I had started this blog and made a privacy policy, and then try again.

Nonsense!

So, I started this blog, I have an occasional reader. It has the right kind of content, I think. I go back to "https://www.google.com/adsense". It is still telling me to refer to the email:

But now I have a new Website! With Privacy Policy! It even has the popup about cookies! So, I enter the website you are now reading and press [Save and Continue].

At first, I think the click doesn't take. My mouse is sometimes buggy that way. Then I notice the little message.

"Cannot connect to the server"? Which server? Mine or one of theirs?

I double- and triple-check my own server. I use a proxy to reach it from the US - no trouble. I try different browsers: no problem. Maybe they're having trouble with the certificate for some reason, even though no else does. Nope: same message for vanilla http.

Maybe it really was their server. I try again an hour later - same problem. A day later. Same.

It can't be their server, not over more than 24 hours. It's not the reachability my server. This message is flat out lying to me.

When all else fails, RTFM - I click on the help link.

Clicking on those two helpful links to google/adsense? You got it: takes me straight back to the original unhelpful page.

Salvation?

I search for others who have had this problem. I find one! Someone who never got their email either. The (Google) Forum says "hey, go to this help location for AdSense, there you can enter your problem. It'll ask for your transaction number from the email, which you don't have, so enter blah-blah instead. You'll get a chat, you can explain the problem, they're very friendly, and you'll be good to go.

There was a reply: "I tried it. It worked. Thanks!" Salvation is at hand!

I click on the link. It redirects:

At this point, you can tell I really don't like bothering people and asking for help. Classic male behavior.

More than a week has gone by - I've been busy.

Sometimes, it helps to have a fresh take.

Two things have occurred to me:

• I haven't registered the blog with Google Webmaster Tools, the way I have with all my other websites.

• What was that stuff off to the right on the help page?

Wish me luck! See the update below.

What, No Snappy Title?

Apparently there are no synonyms for "monetize" and "monetization" - all the top websites claim to have synonyms, but "coinage" and "validation" just don't cut it. And "monetisation"? Really freethesaurus.com?

Update 6-Sep-2016

tl;dr: AdSense would only accept the main site, not the blog subdomain, and I needed a privacy policy on the main site.

That stuff off to the right on the help page that I thought might help:

• Common application problems:
  • enable cookies (already checked)
  • enable SSL 2.0 (already checked)
  • clear cache (already tried)
  • check the email message (not received)
  • check spam folder (duh!)
  • verify correct email entered (already checked)
  • what if I entered the wrong email (it was correct)
  • what if the name doesn't fit (really?)

→ Did I maybe read this and forget it? This is obvious stuff.

• Cancel my application:
  • If you haven't been accepted, then there's nothing to cancel.

→ Okay. Not very helpful to me.

So that still leaves us with that first page: the website content. Maybe, just maybe, it is rejecting the blog out of hand as a sub-domain, and www.softwareschmiede-herndon.de due to lack of privacy policy and/or missing cookie warning.

Adding the cookie-warning turned out to be more of a pain than I thought (scripts working across frames are a bitch), so I ended up reverting to a working version, and just adding a privacy policy. I tried submitting the blog again, which of course failed. Then I tried www.softwareschmiede-herndon.de again.

Hooray! We're a step further!

Sunday night, I get the email. It's in German for some reason, but "Welcome to AdSense"!

It tells me that the first ads will be blank, that I need to put the first ads on the main site, and warns that clicking on the ads myself is a violation of their terms.

So, I placed the first one at the bottom of the start page at https://www.softwareschmiede-herndon.de/

On Monday I get another email, back to English:

Hooray!

The link leads me back to the page where I got the code that is currently appearing as a blank, so I'm leaving it for the moment. If it doesn't start working soon, I'll try regenerating the ad code.

I've added the ad code to the blog; currently at the very bottom of every page.

Lessons

• Read The Friggin' Manual

• Painful as it sometimes is, Read the entire Friggin Manual.

• Sometimes I miss things that have been staring me in the face: "take a step back" from the problem.

Update:

• Perseverance pays.

Resources

• Google "My Business"
• Google AdSense
• Google Webmaster Tools - this is free and has been very helpful.

Footnotes

* You must register with Google as a business to be found in Google Maps or place ads with Google ("Google AdWords"). It was pretty straightforward. Here in Germany you must have a "Gewerbeschein" (license to do business). You will also need: a picture and a logo, a physical location where you receive mail, "opening hours", and a bank account for electronic transfer (a regular bank account will do). They verify the physical address by sending you a postcard with a code on it, and they verify the account by transferring one cent to the account with a code in the remarks. I like it: reasonably secure without being overly onerous.

Copyright © 2016 William David Herndon. All rights reserved.

I was really pretty shocked once Ghost was up and running that it didn't already have comments from the get-go. I mean, what good is a blog without comments?

What now?

I went to the Ghost website and found this page on integrating comments. It mentions that some themes come with comments. I checked that out: the themes that have comments, usually add one of those listed, so I still have that decision.

I love tables. Here's one now:

So, I go with NodeBB.

Preliminaries

I go to NodeBB and find the guide for installing under Ubuntu. I have all the prerequisites, except:

Mongo!

Every time I think of MongoDB, I can't help thinking of Mongo from Blazing Saddles:

(The stunt horse was not harmed - note how the rider tugs on the reigns just before the "punch")

Looking at NodeBB's guide for installing MongoDB, I notice it doesn't match-up with the other info: it mentions Redis as the default, where ghost doesn't even mention Redis as a possibility, and "Note: NPM is installed along with node.js, so there is no need to install it separately" (hah!). So, I just search for the package:

$ apt-cache search mongodb

Reveals a butt-load of packages, so I go ahead and

$ sudo apt-get install mongodb

I check the list: nothing being uninstalled. Among those being installed: mongodb-clients, mongodb-dev and mongodb-server - it looks good. Confirm, and after a few minutes, it's done installing.

Back to the main course

Like with ghost, I made a user (nodebb) and a directory. Then I go ahead and git clone nodebb:

$ cd /var/www/ghost/nodebb
$ sudo su nodebb
$ git clone https://github.com/NodeBB/NodeBB.git nodebb ^**

This, of course, puts nodebb in the directory /var/www/ghost/nodebb/nodebb, not the intended /var/www/ghost/nodebb.
This isn't the first time I've done that. Maybe I should turn this into a script in /usr/local/bin or something:

Note: I need to go su briefly to do this.

$ cd ..
$ mv nodebb temp
$ cd temp
$ mv nodebb ..
$ cd ..
$ rmdir temp

Proceeding with installation per the nodebb guide:

$ cd nodebb
$ npm install --production

That seems to work, again leaving a tree that scrolled everything else off-screen. What's next?

$ ./nodebb setup As in the guide? No!

I tried that: that way leads to pain and suffering (unclear questions, failure to connect to MongoDB). Instead, jump ahead and just start it.

$ ./nodebb start

It starts in the background - I have a prompt again. So, this script works like an /etc/init.d script. I'll have to copy it there and set it up so that it starts on bootup - later.

Now, since it is only available locally (localhost above), I use lynx, which works in a shell:

$ lynx localhost:4567

I confirm the cookie it wants to give me, then get a nice page.  

Then per the ghost instructions, I try to login. Selecting the the [login] link doesn't seem to do anything. I scroll down.

Your browser does not seem to support JavaScript. ...

Of course. Logins usually need JavaScript. Back to plesk. Setup a new subdomain nodebb.softwareschmiede-herndon.de. Redirect the new subdomain to localhost:4567 using ProxyPass (see My Oddysey to Blogging-Land).

Then, opening a regular browser to nodebb.softwareschmiede-herndon.de, I am greeted by this glorious sight:

Note: I reconstructed the process in VirtualBox, which is why it says localhost:4567 instead of nodebb.softwareschmiede-herndon.de

For administrator, choose the username you want your users to see when you post - changing it later can cause you problems (yes, that's experience talking).

Choose a strong password right from the get-go: if you don't, it will fail without warning you: it won't create your admin account, while otherwise completing installation. You will be locked out.

If you do manage to lock yourself out, remove the config.json in the nodebb folder and try again - that worked for me.

Of course, change the database to MongoDB.

Then click "Install NodeBB". On the following page click "Launch NodeBB". Then login: you should land on the admin page.

Finally with setup done, I now install the "comments widget" as per the ghost instructions (this page again):

$ npm install nodebb-plugin-blog-comments

The instructions say to "Click on the Reload button at the bottom of the main Admin page". It's called the "Dashboard" now and restart button is:

Activating the plugin:

• "open up Extended -> Plugins and activate nodebb-plugin-blog-comments". It's "EXTEND" now, but fine. I click on Activate for nodebb-plugin-blog-comments.
• then I also activate nodebb-plugin-dbsearch - I want comments to be searchable.
• Return to the Dashboard
• Click [RESTART]
• Refresh your browser (usually F5) - Do not skip this step: there is no auto-refresh when you click [RESTART], and the menu you need for the next step will not be there.

Configuring the plugin:

• "Go to Installed Plugins -> Blog Comments".
• I fill in the page and click

The promised button with the word "Save" on it in their picture has morphed into this badly drawn image of a media that kids these days have never actually seen. The inner blue circle needs to be centered, it's the center of rotation for the disk inside! And there's no mouseover text - no hint! I have blind friends for whom this kind of thing matters! Sorry, I digress.

• Go back to the Dashboard and
• click [RESTART] again.

Inserting the comments into ghost - as per the instructions:

• I find the file post.hbs - /var/www/ghost/content/themes/casper/post.hbs
• I open post.hbs in a text editor
• find the location of {{content}}

• Copy and paste the following text after that line:
<a id="nodebb-comments"></a> <script type="text/javascript"> var nbb = {}; nbb.url = 'http://nodebb-forum.com'; // EDIT THIS   (function() { nbb.articleID = '{{../post.id}}'; nbb.title = '{{../post.title}}'; nbb.tags = [{{#../post.tags}}"{{name}}",{{/../post.tags}}]; nbb.script = document.createElement('script'); nbb.script.type = 'text/javascript'; nbb.script.async = true; nbb.script.src = nbb.url + '/plugins/nodebb-plugin-blog-comments/lib/ghost.js'; (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(nbb.script); })(); </script> <script id="nbb-markdown" type="text/markdown">{{{../post.markdown}}}</script>

• Edit the line with EDIT THIS, in my case, changing nodebb-forum.com to nodebb.softwareschmiede-herndon.de.
  Note: I first tried to be smart and hide nodebb, using localhost:4567. Of course it doesn't work - this script is executing in the client machine. Yay sleep deprivation!

• I restart ghost:
  $ sudo /etc/init.d/ghost restart

I go to my first blog article at the bottom, note that it says "Read only". Oh, but there's this nice button: [PUBLISH THIS ARTICLE TO NODEBB]. I press it, and voila: I can comment!

I'm not done yet.

As a good programmer, I test it from the users perspective: I register a new user and make a comment. Hooray!

Two days later, a friend skypes me with a comment. I tell him to go ahead and post it. He says he can't yet - apparently there's a one hour waiting period after registration. He thinks. It's not really clear to him.

I'm not sure why it worked straight away with my test user. Maybe some cookie from my admin login was still active?

In any case, I'm not happy with my users having to wait an hour by default:

• I go to nodebb.softwareschmiede-herndon.de
• I click on the Admin icon
• I select the menu Settings→User
• I activate Email registration.
• I click the funky save icon.
• I change Account Lockout Duration to 1 minute.
• While I'm at it, I go to Settings→Email and set that up, too.

I'm eventually going to have to take spam counter measures - some sort of challenge against bots. But for now, I'm done.

Lessons

• Being bleeding edge is exhausting.
• Sleep is good - coffee can only do so much.
• When testing, use a different browser without your cookies. Better yet, a different machine. Still better, if possible, have someone else try it - you might have developed error avoidance habits.
• Having a spare Ubuntu in a VirtualBox is very useful when trying to reconstruct what you did from incomplete notes.

Resources

• How to enable comments on a Ghost blog
• NodeBB / Guide for installing NodeBB under Ubuntu.
• Lynx - the text-only browser
• MongoDB.com - warning: it is a flashy commercial site. All the mongodb.org sites now redirect to their mongodb.com equivalents. *sigh*
• Innotek's Sun's Oracle's VirtualBox

^* Why you should never use mongodb, by Sarah Mei, a contributor to Diaspora. Diaspora was founded as a more privacy rights aware alternative to Facebook, but got off to a bad start and has yet to take off.

^** I skipped the -b v1.x.x recommended in the git command: I'd have to look up the most recent versions, decide which is stable enough, which might involve looking into individual bugs, etc. Nah, I want this done. We're already living dangerously getting a bleeding edge BB, we'll just get the bleeding edge version.

Copyright © 2016 William David Herndon. All rights reserved.

So, of course I login to my "Parallels^® Power Panel":

You can ignore the red warning triangle clearly visible at the top: I complained to my hoster years ago: they told me it meant nothing, but if it bothers me, I should contact Parallels^®. I found it difficult back then to do, and so quickly gave up over so trivial a matter.

Taking most of a day to reinstall something that should take minutes warrants a larger time investment this time around. They really should fix this.

If you have sharp eyes, you can see in the upper right hand corner, that the copyright extends to 2012. Probably good enough to complain to my hoster, but I want to know more precisely. It should be a simple matter, right?

The menus turn up nothing. The controls in the menus turn up nothing. Duckduckgo-ing* turns up nothing.

Well, I think I finally found it: click on "Status Changes" or "Resource Alerts", and then look at the hint message area under "Log Out". Do not move the mouse over a menu though, otherwise the version, "04.01.00.00.47", will be replaced with a helpful hint for the menu.

What kind of software producer doesn't want you to know the version? They usually love to tell you to just update to the latest version to fix all your problems.

Whatever, now armed with the version number, I'm off to try and find out what the latest version is, so I can ask for it.

Except Parallels^® "Power Panel" doesn't seem to exist as a product anymore. Not on their website anyway. All I found was some documentation that looks older than what I already have.

I guess that it got spun off or something - an existing company wouldn't just abandon a successful product, would they?

I go back to my hoster: they seem to use a lot of terms interchangeably, that aren't entirely clear to me. The buzz words are "Virtuozzo", "Odin", "VZPP" (= "Virtuozzo Power Panel"), and, of course, Parallels^®. All of the documentation I can find just assumes you already know about them.

So, let's be a little more systematic:

Back up. What was the title of that news article?

"After Parallels Spin Out, Virtuozzo Refocuses on Partners and Technology".

"Virtualization platform provider Virtuozzo spun off as a standalone company from Parallels about six months ago as Odin was acquired by Ingram Micro."

Mergers, spinoffs, and acquisitions. This is not conducive to smooth transitions. But it looks like I've found my answer: the current equivalent of Parallels^® (Power Panel) is Virtuozzo Containers™ (whatever they call their panel now). With the renaming, it is highly unlikely that there is a direct compatible upgrade path. Crap.

So, I've just spent a good many hours finding out that there is no one to complain to: the original manufacturer doesn't exist anymore, and the hoster doesn't upgrade because they can't.

Take all of this with a grain of salt: I haven't actually contacted anyone to confirm. To do that properly would probably take more hours, perhaps days, seeing as how all parties concerned seem to be obfuscating the truth. And I'm not willing to invest that much at this time.

*Yeah, DuckDuckGo is my default instead of Google. I'm not comfortable with Google knowing more about me than I do. I still use Google when DuckDuckGo doesn't give me the results I need.

Lessons

• This is one of the reasons I prefer Open Source: sometimes software doesn't fit in the business plan, and suddenly you're stuck with orphaned software that you cannot fix yourself.

Resources^**

• HostEurope - a decent hoster in Germany.
• Strato - another decent hoster in Germany, for parity.
• Plesk - WebAdmin tool that works well with Power Panel
• Parallels - Original Power Panel maker.
• Odin - Original maker of Plesk? Now belongs to Ingram Micro.
• Virtuozzo - Maker of current Virtuozzo Containers™.
• Docker - Maker of a competing "containerization" software.

^**Though most all of these resources are for profit businesses, I have received no money for this article or their mention. If I were them, I wouldn't give me money for this either: this article is not exactly what I'd call an endorsement.

Related blog posts

• My Odyssey to Blogging-Land
• How to: Setting up a VPS (not yet written)

Copyright © 2016 William David Herndon. All rights reserved.

Which blogging software should I use?

After deciding to create a blog, the next question is how. I could go to blogger or some other hoster, which I've done before, but that's no fun!
So, if I'm doing this myself, the next question is, which blog software / Content Management System to use? A little bit of searching and winnowing, and I came up with this list:

Name	Logo	Language	License	Familiarity
WordPress		PHP	GPLv2	very
Joomla		PHP	GPL	some
Drupal		PHP	GPLv2	some
TYPO3		PHP / JavaScript*	GPL	none
Ghost		node.js	MIT	none

* Note that JavaScript is now officially called ECMA.

node.js is a shiny new language platform from Google Linux Foundation (sorry! Somehow got it confused with Google's JavaScript based Go) that people either swear by or at. It is written in JavaScript. Which is itself written in C/C++. They're piling the layers on really deep these days.

Apparently I like tearing my hair out. Also, I need shiny content for this blog. I choose Ghost.

Installing ghost blogging software

A "short" synopsis:

• Go to ghost.org
• Find installation instructions for Linux
• Use putty to connect to my Linux VPS.
• Installing node.js:
  • Try sudo apt-get install node - notice it's the wrong package before confirming.
  • Search for the right package (I could write a lot about dealing with apt/dpkg)
  • sudo apt-get install nodejs
• Make the user "ghost", the directory "/var/www/ghost", chown it, cd to it, curl the zip file and unzip it.
• npm start --production → npm: command not found.
• Installing npm:
  • Look up npm: "package manager for node.js", sounds harmless.
  • sudo apt-get install npm and confirm.
  • Wait!!! It's uninstalling plesk?!?! The VPS management software my host provides?!
  • I try to cancel, but it's having none of it. Before it scrolls beyond range, I manage to copy the list of the packages it's uninstalling.
  • I verify that plesk is not working, though my websites are working.
  • I spend the rest of the day and evening into the wee hours of the next morning recovering plesk enough to be usable. It's still not completely recovered, but good enough for now. I need to write to npm maintainers about this conflict, and I need to write a long letter to Parallels about the (un)usability of their "Power Panel" (which I used to recover) spend a few useless hours finding out that there is no one to report to, then write a blog post about it.
  • I still have the problem of installing npm without uninstalling plesk. Some searching finds me Tobi's Software Development Blog.
    • Method 1 simply does not work: it still wants to uninstall plesk.
    • Method 2: curl https://www.npmjs.com/install.sh | sh requires privileges, and is executing a script, sight unseen...
    • So, I download the script, examine it, it looks okay, I execute it. → npm cannot be installed without Node.js
    • But, node.js is installed. After some script debugging, I realize it is looking for "node", whereas the program is nodejs.
    • I replace "node" with "nodejs".
    • The script generates some sort of tree that scrolls off the top of the terminal, followed by "It worked." That is bad form, but okay.
• I go to /var/www/ghost and execute npm install --production, as per instructions. → node: no such file or directory.
  • The node / nodejs problem is apparently everywhere. I add a link to fix it:
    • cd /usr/bin
    • sudo ln -s nodejs node
• Again I go to /var/www/ghost and execute npm install --production. This time a tree scrolls off the screen with no final message - it appears to have worked.
• First time start up:
  • I execute npm start --production → it appears to be running. In the foreground, as expected.
  • Neither www.softwareschmiede-herndon.de:2368, nor 5.35.246.86:2368 is working from my home browser.
  • I check if the firewall might be blocking it: it turns out the firewall is down since the npm fiasco (above). I put off fixing that (good choice: a five hour excursion, as it turned out).
  • Since this might be an access problem, I open another putty session and use lynx to localhost:2368. → At first an error page (hey, something is running!), but a simple refresh and: success!
  • A little searching shows that the problem accessing from the browser is that the listening port is set to 127.0.0.1: I change it to 0.0.0.0 → I can access it at www.softwareschmiede-herndon.de:2368
  • I create my admin user.
  • I reconsider the setup: a more typical setup would actually redirect from blog.softwareschmiede-herndon.de to localhost:2368, so that search engines can find it. Using the port 2368 directly should be discouraged or impossible.
• Setting up the redirect for http://blog.softwareschmiede-herndon.de/
  • I create the plesk subdomain
  • After much searching, I find that the "Hosting type" "forwarding" seems to do what I want.
  • After much testing, I find forwarding does not do what I want: either it tells the browser a new address or it inserts it all in a frame, both of which are really bad for search engines.
  • I have a minor quest through the depths of nginx, which was not installed and when installed, made all my other websites unavailable. Uninstalled again.
  • Apache really should have options for this, so why are all the solutions pointing elsewhere? I look specifically for an Apache solution, and find one. With a little more work, I get it to work in plesk:
    • Create a normal subdomain
    • Go to the "Apache & nginx Settings" for the subdomain.
    • Under additional directives for HTTP (and HTTPS) enter:
    • RewriteEngine On
    • ProxyPass / http://localhost:2368/
    • ProxyPassReverse / http://localhost:2368/
• I set ghost up as a service:
  • I create a script in /etc/init.d to start ghost as a service.
  • I create a conf file, /etc/init/ghost.conf
• So, finally working, I make an Introduction post.
• I want to test commenting... but there's no comments!
  • How I got comments working is planned as another post.

Lessons

• Never assume that anything that should be short and simple actually is. But you already new that.
• Always, no matter how innocuous the install, review the output of apt-get carefully before confirming.
• Script debugging, bash -x, is your friend.
• When developing your own package:
  • Look out for name conflicts before they become trouble.
  • Regularly test installing on various systems, to be sure things aren't broke for beginners.
• I need to report problems, so that they get fixed and others aren't bit by them.

Resources

• ghost.org / Installation instructions for Linux
• Latest ghost zip
• Tobi's software development blog / Install nodejs and npm with plesk
• /etc/init.d/ghost

Related blog posts

• Virtualization Crashes into Business Reality
• Welcoming Comments - Installing NodeBB comments on ghost
• Monetizing my blog (not yet done, much less written)

Copyright © 2016 William David Herndon. All rights reserved.