Moving on from “should we do it?” (with the answer to most real-world scenarios being “yes, and as a bonus it can help block a lot of spambots“), here’s how to restrict several Internet services — Nginx, Apache, Postfix, and Dovecot — to TLSv1.2 or newer.
As usual, these are not complete guides for any of those servers; I’m assuming you already have them working fine (including TLS encryption), and just want to disable any TLS protocols lower than v1.2. (If you need to add TLS to a non-TLS server, see instructions for Nginx, Apache, Postfix, and Dovecot.)
Nginx:
In each virtual host’s server section — or, even better, if you’re using Let’s Encrypt, in /etc/letsencrypt/options-ssl-nginx.conf or its equivalent –, add the following (or replace any existing ssl_protocols entry):
ssl_protocols TLSv1.2 TLSv1.3;
Restart Nginx, and test it on SSL Labs. You should get something like this1:
(Note: if you’re using a very old version of Nginx, it may not accept the “TLSv1.3” parameter and refuse to start; in such a case, remove it — or, better yet, upgrade your system. 🙂 )
Apache:
Similarly to Nginx, you can add one of the following to each Virtual Host, or to the global HTTPS configuration (typically in /etc/httpd/conf.d/ssl.conf), or, if using Let’s Encrypt, to /etc/letsencrypt/options-ssl-apache.conf :
SSLProtocol +TLSv1.2 +TLSv1.3
or:
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
Right now, they’ll do the same thing: allow TLSv1.2 and v1.3 only. Personally, I like the second version (which disables older protocols) better, for two reasons: 1) it’ll work even with some ancient Apache version that doesn’t recognize “TLSv1.3”, and 2) when future TLS versions are added, they’ll be enabled, making it more future-proof.
Again, you can test the new configuration on SSL Labs.
Postfix:
Add the following to /etc/postfix/main.cf (replacing any equivalent entries, if they exist).
After restarting Postfix, you can test its available protocols with Immuniweb’s SSL Security Test (specify something like yourhostname:25, or yourhostname:465 if not using STARTTLS).
Dovecot:
Add the following (or replace it if it exists) to your SSL configuration (typically /etc/dovecot/conf.d/10-ssl.conf):
ssl_min_protocol = TLSv1.2
Restart Dovecot, and test it on Immuniweb’s test (use something like yourhostname:143 , or yourhostname:993 if not using STARTTLS).
TLSv1.0, used not only for HTTPS but also for secure SMTP, IMAP, etc., has been around for a while — 1999, in fact –, and, while not “hopelessly broken” like SSL 2 and 3 are, there have been many successful attacks/exploits against it in the past two decades, and while most implementations (in terms of both servers and clients) have been patched to work around those, there may always be another one just around the corner. Also, TLSv1.0 came from a time where it was desirable to support as many different ciphers as possible, which means that, by default, some ciphers later found to be insecure may still be enabled.
Therefore, the end of TLSv1.0 has been talked about for a while, and, indeed, the main browser vendors — Google, Mozilla, Apple, and Microsoft — have announced that they’ll deprecate TLSv1.0 (and 1.1, which is little more than a revision of 1.0) by early 2020.
Now, until a few days ago I still had TLSv1.0 enabled on my servers (including Nginx, Postfix and Dovecot); my belief then being that some encryption was always better than no encryption, and besides we were mostly talking about public websites, that didn’t ask for user credentials or anything. Furthermore, any modern browsers/email clients default to secure protocols (TLSv1.2 or better), so there would never be, “in real life”, a situation where important/sensitive data (say, me logging on to my own mail server, or to one of my WordPresses) was being transmitted in an insecure way.
On the other hand… doesn’t the above mean that, “in real life”, nobody will really be using TLSv1.0 “legitimately”? Doesn’t it mean that setting up TLSv1.2 as a minimum will have no adverse consequences?
So I investigated it a little:
in terms of browsers/clients, requiring TLSv1.2 or above means: 1) no Internet Explorer before version 11; 2) no Safari before 6.x; 3) no default Android browser before Android 4.4.x (but those users can still use Chrome or other non-default browsers). (You can check this out on SSL Labs). I don’t have a way to check email clients that old, but you can assume that anything released after the browsers just mentioned will be OK;
in terms of my own server logs (mostly Nginx, but also Postfix, in terms of opportunistic encryption), apparently, only spambots and such are using TLSv1.0; any “legitimate” email servers use TLSv1.2 or 1.3. The same thing with my websites — any TLSv1.0 entries in my logs are always from some bot (and not one for a “well-known” search engine), not actual visitors. 1
In short: you can probably move to requiring TLSv1.2 or above now (and, indeed, I’ve done so since a few days ago), unless you’re in some very peculiar situation — maybe your site needs to be accessed by users of a company whose IT policy includes not using software newer than 20 years old? Or you have a CEO who’s always refused to upgrade his beloved Symbian phone, but who’d flip out if he couldn’t access your company’s webmail with it? 🙂 Otherwise, my advice is: go TLSv1.2 and don’t look back.
However, and since this post is already quite long, I’ll leave the specific instructions for disabling TLS before 1.2 in Nginx, Apache, Postfix, and Dovecot for the next one…
For extra fun, instead of Debian/Ubuntu, we’ll be using Red Hat/CentOS (more precisely, CentOS 7.6 with the latest updates as of January 10th, 2019, but this shouldn’t change for any other 7.x versions of either CentOS or RHCE). And we’ll keep firewalld and SELinux enabled 1, even though, as far as I know, in most companies (well, at least the ones I’ve worked at) it’s, typically, official policy to disable both. 🙂
So, let’s assume you already have a CentOS/RHEL machine, with internet access and the default repositories enabled. First, we’ll enable the EPEL repository (to provide Let’s Encrypt’s certbot):
Now we’ll install certbot (which will also install Apache’s httpd and mod_ssl as dependencies) and wget (which we may need later):
yum install python2-certbot-apache wget
To be run certbot and allow it to verify your site, you should first create a simple vhost. Add this to your Apache configuration (e.g. a new file ending in .conf in /etc/httpd/conf.d/):
You should now have the virtual host configured in /etc/httpd/conf.d/ssl.conf . Let’s add a few lines to it (inside the <VirtualHost></VirtualHost> section):
Header always set Strict-Transport-Security "max-age=15768000"
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384::ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384
SSLCipherSuite TLSv1.3 TLS_CHACHA20_POLY1305_SHA256:TLS_AES_256_GCM_SHA384
SSLHonorCipherOrder on
SSLCompression off
(Don’t mind the “TLSv1.3” line for now, it won’t be used — if at all — until later in this post, and won’t have any effect if it isn’t.)
This should get you an A+ rating, with all bars at 100% except for Key Exchange. A good start, right? 🙂 However, that final bar can be a bit tricky. The problem here is that SSL Labs limits the Key Exchange rating to 90% if the default ECDH curve is either prime256v1 or X25519, the first of them being the OpenSSL default… and Apache 2.4.6 (the version included in all 7.x RHELs/CentOSes) doesn’t yet include the option to specify a different list of curves (of which secp384r1 will give a 100% rating, and won’t add any compatibility issues).
Assuming you don’t want to install a non-official Apache, then the only way (as far as I know) to specify a new curve is to add the results of running:
openssl ecparam -name secp384r1
to your certificate file. But Let’s Encrypt certificates are renewed often, so you may need some trickery here — such as adding something like this to root’s crontab:
Alternatively, you can upgrade Apache to a newer version. CodeIT (for instance) provides replacement packages for RHEL and CentOS, which you can use by doing:
However! This will still not give 100%, since CoreIT’s Apache appears to come with a newer OpenSSL (1.1.1) statically linked to it, which means it’ll enable TLS 1.3 by default. And there’s nothing wrong with it… except that, from all the tests I’ve performed, Apache ignores the specified order of the ECDH curves when using TLS 1.3. With the line above, it’ll use secp384r1 for TLS 1.2, but X25519 for TLS 1.3 — and that’s enough to lower the Key Exchange bar to 90% again. 🙁
Given this, there are now several options:
1- disable TLS 1.3 (it’s still relatively new, and most people are still using 1.2): change the line:
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
to:
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1 -TLSv1.3
2- force secp384r1: change the “SSLOpenSSLConfCmd Curves” line to simply:
SSLOpenSSLConfCmd Curves secp384r1
3- ignore this situation altogether: X25519 is considered to be as secure as secp384r1, if not more; even SSL Labs mentions not penalizing it in the future in their (early 2018) grading guide — they just have been a bit lazy in updating their test. It’s quite possible that a future update of their tool will give a full 100% score to X25519.
Anyway, most of this was done for fun, for the challenge of it. 🙂
This post should be titled “How to get a 100% score on SSL Labs (Nginx, Let’s Encrypt) as of April 2018“, since SSL Labs‘s test evolves all the time (and a good thing it does, too.) But that would be too long a title. 🙂
The challenge:
As we’ve seen before, it’s relatively easy to get an A+ rating on SSL Labs. However, even that configuration will only give a score like this:
Zurgl.com’s current SSL Labs score
Certainly more than good enough (and, incidentally, almost certainly much better than whatever home banking site you use…), but your weirdo of a host can’t look at those less-than-100% bars and not see a challenge. 🙂
First, note that all of the following will be just for fun, as the settings to get 100% will demand recent browsers (and, in some cases, recent operating systems/devices), so you probably don’t want this in a world-accessible site. It would be more applicable, say, for a webmail used by you alone, or by half a dozen people. But the point here isn’t real-world use, it’s fun (OK, OK, and learning something new). So, let’s begin.
Getting that perfect SSL Labs score:
For this example, I’ll be using a just-installed Debian 9 server, in which I did something like:
apt -y install nginx
echo "deb http://ftp.debian.org/debian stretch-backports main" >> /etc/apt/sources.list.d/stretch-backports.list # if you don't have this repository active already
apt -y install python-certbot-nginx -t stretch-backports
(The reason for the above extra complexity is the fact that the default certbot in Debian Stretch is too old and tries to authenticate new certificates in an obsolete way, but there’s a newer one in stretch-backports.)
I also created a simple index.html file on /var/www/html/ that just says “Hello world!”. Right now, the site doesn’t even have HTTPS.
So, let’s create a new certificate and configure the HTTPS server in Nginx:
Testing it on SSL Labs, it gives… exactly the same bars as in the image above, but with an A rating instead of A+. Right, we need HSTS to get A+ (and currently certbot doesn’t support configuring it automatically in Nginx), so we add this to the virtual host’s server section:
(Note: remove “includeSubDomains;” if you have HTTP sites on any subdomain. Also, HSTS makes your site HTTPS-only, so you can’t use it if you want to keep an HTTP version — in which case you can’t get better than an A rating.)
This upgrades the rating to A+, as expected, but the bars still didn’t change. Let’s make them grow, shall we?
Certificate:
It’s already at 100%, so yay. 🙂
Protocol Support:
Just disable any TLS lower than 1.2. In this case, edit /etc/letsencrypt/options-ssl-nginx.conf and replace the ssl_protocols line (or comment it and add a new one) with:
ssl_protocols TLSv1.2;
WARNING: any change you make to the file above will affect all virtual hosts on this Nginx server. If you need some of them to support older protocols, it’s better to just comment out that option in that file, and then include it in each virtual host’s server section.
EDIT: actually, it turns out that ssl_protocols affects the entire server, with the first option Nginx finds (first in /etc/nginx.conf, then in the default website (which possibly includes /etc/letsencrypt/options-ssl-nginx.conf)) taking precedence. So just choose whether you want TLS 1.0 to 1.2 (maximum compatibility) or just 1.2 only (maximum security/rating); if you need both, use two separate servers.
Key Exchange:
We’ve already created the new certificate with a 4096-bit key (with “—rsa-key-size 4096“, see above), and according to SSL Labs’ docs this should be enough… but it isn’t. After some googling, I found out that you also need to add the following option inside the server section in Nginx:
ssl_ecdh_curve secp384r1;
Since the default Nginx+OpenSSL/LibreSSL setting, either “X25519” or “secp256r1” (actually “prime256v1“), also lowers the score.
EDIT: again, ssl_ecdh_curve affects the entire server, so you can’t use different default curves for each virtual host. So I’d suggest (in /etc/letsencrypt/options-ssl-nginx.conf):
ssl_ecdh_curve secp384r1:X25519:prime256v1;
if you want the 100% rating, or:
ssl_ecdh_curve X25519:prime256v1:secp384r1:
otherwise. This one doesn’t affect compatibility, by the way; it’s just a question of the preferred order.
The certificate’s key size (4096 or 2048) is, like the certificate itself, specific to each virtual host.
Cipher Strength:
For 100% here, you need to disable not only any old protocols, but also any 128-bit ciphers. I started from Mozilla’s SSL tool (with the “Modern” option selected), then removed anything with “128” in its name, then moved ChaCha20 to the front just because, and ended up with this line, which I added to /etc/letsencrypt/options-ssl-nginx.conf (replacing the current setting with the same name):
(This time, this setting can indeed be specified for each virtual host, though you’d need to comment it out in the file above (which affects all virtual hosts with Let’s Encrypt certificates) if you want to do so, otherwise they’ll conflict.)
The result:
As said near the beginning, this is not something you likely want to do (yet) for a production, world-accessible site, as it will require relatively modern browsers, and you typically can’t control what visitors use1. According to SSL Labs’ list, it’s not actually that bad: all versions of Chrome, Firefox, or Safari from the past couple of years are fine, but no Microsoft browser older than IE11 will work, nor will Android’s default browser before 7.0 (Chrome on Android, which is not the same thing, will do fine.) I’d suggest it for a site where you can “control” the users, such as a small or medium-sized company’s webmail or employee portal (where you, as a sysadmin, can and should demand up-to-date security from your users).
But the main point of this exercise was, of course, to see if I could do it. Challenging yourself is always good. 🙂 And if you can share what you learned with others, so much the better.
If you normally browse the web (and you don’t block ads), you’ve probably already seen hundreds, if not thousands, of advertisements for “VPNs” or “VPN software/subscriptions” — how you “need” one to have any kind of privacy, how “they”1 can track you everywhere if you don’t use one, and so on. And they’re usually not cheap. Interestingly, and probably because of some recent “revelations”, all those advertisements seem to focus on privacy or anonymity only.
But using a VPN can offer you something else: mobile security; namely, the ability to make your mobile devices “be”, on demand, part of your home network (or your server’s network, if you have, say, a VPS somewhere), no matter where you are, regardless of whether you’re using 3G/4G mobile data, or some random WiFi hotspot. Yes, even one with no encryption at all; it won’t matter. You can, absolutely, trust random open hotspots again; even their owner won’t be able to read or alter your traffic in any way. And you can do it for free, too (assuming you already have an always-connected Linux server); you just need to configure your own OpenVPN server, which you’ll be able to do by following this (hopefully accessible) guide.
(Yes, you read the title correctly. For extra fun, and to prevent this blog from being too focused on Ubuntu/Debian, this time I’ll be using Red Hat Enterprise Linux / CentOS (and, I assume, Fedora as well.) Later on, I may post a Debian-based version.)
Configuring a basic HTTP site on Nginx
(Note: if you already have a working HTTP site, you can skip to the next section (“Adding encryption…”))
Yes, the post title mentions an “existing” website, which I believe will be the case in most “real world” situations, but installing a new one is actually very easy on CentOS1. First, do:
yum -y install epel-release; yum -y install nginx
Then create a very basic configuration file for the (non-HTTPS) site, as /etc/nginx/conf.d/mysite.conf :
Then, of course, create the /var/www/mysite directory (CentOS doesn’t use /var/www by default, but I’m far too used to it to change. 🙂 ) If you’d like, create an index.html text file in that directory, restart nginx (“service nginx restart” or “systemctl restart nginx“, depending on your system’s major version), and try browsing to http://mysite.mydomain.com . If it works, congratulations, you have a running web server and a basic site.
Adding encryption to the site (not using Let’s Encrypt):
Second, edit the site’s configuration file (in the “starting from scratch” example above, it’s “/etc/nginx/conf.d/mysite.conf“), and copy the entire server section so that it appears twice on that text file (one after the other). Pick either the original or the copy (not both!), and, inside it, change the line:
listen 80;
to:
listen 443 ssl http2;
(Note: the “http2” option is only available in Nginx 1.9.5 or newer. If your version complains about it, just remove it, or upgrade.)
This should be enough — restart Nginx and you should have an HTTPS site as well as the HTTP one.
And what if you want to disable HTTP for that site and use HTTPS only? Just edit the same configuration file, look for the server section you didn’t change (the one that still includes “listen 80;“), and replace the inside of that section with:
(replacing “mysite.mydomain.com” with yours, of course.)
Answer the questions it asks you: a contact email, whether you agree with the terms (you need to say yes to this one), if you want to share your email with the EFF, and finally if you want “No redirect” (i.e. keep the HTTP site) or “Redirect” (make your site HTTPS only).
And that’s it (almost — see the next paragraph) — when you get the shell prompt back, certbot will already have reconfigured Nginx in the way you chose in the paragraph above, and restarted it so that it’s running the new configuration. You may want to add “http2” to the “listen 443 ssl;” line in the configuration file (it’ll probably be the default someday, but as of this post’s date it isn’t), and don’t forget your options for improved security and security headers.
Only one thing is missing: automatically renewing certificates. Strangely, the certbot package configures that automatically on Ubuntu, but not on CentOS, from what I’ve seen (please correct me if I’m wrong). The official Let’s Encrypt docs recommend adding this (which includes some randomization so that entire timezones don’t attempt to renew their certificates at precisely the same time) to root’s crontab:
(Note: It’s possible to use Let’s Encrypt to create ECDSA certificates, but as of this writing you have to do most of the work manually (creating a CSR, etc.), and you lose the automatic renewal, so for the moment I suggest using RSA certificates. I hope this changes in the future.)
(Welcome to “Note to Self”, a new series of hopefully short posts where I attempt to remind my future self of the correct way of doing things I often sometimes get wrong, even after decades.)
The correct syntax for specifying nameservers in /etc/resolv.conf is:
nameserver 1.2.3.4
not:
server 1.2.3.4
Worse, the system won’t complain; it’ll just ignore those entries, so you may wonder for a while about why name resolution is not working.
Of course, when adding an entry to an existing file, you probably won’t make this basic mistake, as any current entries will use the correct “nameserver” syntax. But when creating a new file from scratch… not that it happened to me yesterday, oh no. 🙂
SPF, or Sender Policy Framework, is a way for domain owners to say: “these are the email servers my domain sends mail from; anyone else is attempting to impersonate me.” These days, a lot of email services implement it in some way (both for receiving and sending email), and, if you administer your own server, it’s quite easy to implement, as we’ll see.
There are actually two parts to SPF: configuring it for your domain (so that your outgoing emails have a better chance of being accepted and emails impersonating your domain have a better chance of being refused), and configuring your email server to check whether incoming emails come from an SPF authorized server or not. Without further ado:
1- Configuring SPF for your domain’s outgoing mail:
This is actually done in DNS, not in the email server. Simply add a TXT record for your domain, with something like this:
There are other configuration parameters, such as getting the servers from the domain’s MX record (add “mx“), and copying another domain’s authorized servers (“include:otherdomain.com“). And, of course, if you only have one server sending mail, you don’t need two “a:” entries.
Pay special attention to the “-all” part at the end: that instructs other servers to refuse any email apparently coming from your domain that isn’t authorized for it. That should ideally be the final configuration, but if you’re doing this for the first time you may want to change it to “~all“, which means “softfail”: it suggests to the other side that the mail is probably not legit, but that it shouldn’t refuse it outright because of it. After testing it for a while, if you see no problems then you can change it to “-all“.
2- Configuring SPF checking for incoming mail in Postfix:
I’ll assume you already have a working Postfix server (hopefully already using TLS encryption, hint hint), and that you want to respect any “-all” configurations (see above) in incoming mail — in other words, if a domain says “all my mail comes from server X, refuse anything else” and you receive an email purportedly from that domain, but from a server Y, it’s refused (for “~all“, it’d generate a warning).
First, install pypolicyd-spf. On a Debian-based system, the package (as of this writing) still has its old name, so enter the following:
apt install postfix-policyd-spf-python
Edit /etc/postfix/master.cf, and add:
# spf
policyd-spf unix - n n - 0 spawn
user=policyd-spf argv=/usr/bin/policyd-spf
Then edit /etc/postfix/main.cf, and look for the “smtpd_recipient_restrictions” section. That section will probably begin by accepting mail from your networks and from authenticated users, then rejecting non-authorized relaying, possibly followed by some white- or blacklists, and maybe a couple of filters, finally ending with “permit“. Before that “permit“, add:
check_policy_service unix:private/policyd-spf,
(note the comma at the end; without it, Postfix wouldn’t read the rest of that section.)
IMPORTANT: while you’re still in testing, it may be a good idea to add “warn_if_reject” to the beginning of the previous configuration line. With it, Postfix will still log SPF failures, but won’t actually reject an email because of it. When you’re certain everything is fine, you can then remove that prefix (and then reload Postfix, of course).
Add also the following to that configuration file, outside of any section:
policy-spf_time_limit = 3600
And you’re done. Reload or restart Postfix, and check your logs to see if everything is working as intended.
Note that other tools, such as SpamAssassin, may also use SPF to weigh whether an incoming mail is more or less likely to be spam. That, however, is unrelated to the Postfix configuration above, which does a binary “block/don’t block”.
Like any web server, Nginx logs all accesses/hits by default, and if you have some kind of log-based analytics tool (such as AWstats) you probably have it already set to ignore hits from the host itself (usually by skipping both localhost (127.0.0.1) and the server’s public IP address). If you don’t do so, then your statistics will probably be inflated (“wow, my newly created site is surprisingly popular!“), and, unless your site has many regular users, the IP address at the top of the “visitors” table will almost certainly be your own. Local hits can come from several culprits: for instance, some web software such as WordPress or MyBB use special URLs as a form of cron replacement, and also you may have yourself be monitoring your site in some way (such as, in my case, benchmarking a particular URL and MRTGing the access speed).
OK, so your analytics don’t show it anymore — but what about the log files themselves? Maybe you don’t want a large percentage of them being composed of internal hits (especially in the case of those “let’s measure the average response speed for 100 hits every 5 minutes” benchmarks, which for that site made up some 95% of its access logs…). Maybe it even interferes with some other tools you’re using, such as something like DenyHosts or Fail2ban, to detect some abuse patterns, even after whitelisting your external IP. How about having the option not to log them at all? 1
On Nginx, this is actually pretty easy to do.
1- in the Nginx main configuration file, add the following:
Replacing YOUR_EXTERNAL_IP with… you can probably guess. 🙂
Important: if your virtual hosts’ configurations are in another directory that is “included” from nginx.conf, take care to add the above before the include lines (e.g. include /etc/nginx/conf.d/*.conf) . Otherwise, the virtual hosts won’t have the “$notlocal” variable defined yet, and Nginx won’t start because of it.
2- for each virtual host where you want to stop logging local hits, edit the access_log line, changing it from something like:
Note the addition of “combined“. “combined” is the default format for Nginx access logs, so specifying it isn’t usually needed, but apparently if you want to specify the “if” condition, it must come after the log format option (otherwise, I guess Nginx thinks you want a log format of “if“, which of course doesn’t exist and will prevent the server from starting.)
That it! Restart Nginx, and enjoy your much cleaner/smaller logs.
Some time ago, I wrote about how to set up Nginx to get an A+ rating on SSL Labs (focusing specifically on an ECDSA certificate, but you can certainly achieve the same with a (more common) RSA certificate). SSL Labs' test is very comprehensive, warning the user of most potential problems, including SSL/TLS implementation bugs (update your software!), misconfigurations, problems with the certificate itself, etc.. But there is another similar tool, securityheaders.io, which focuses on something different: security-related headers. Such headers are sent by the web server (such as Nginx or Apache) and affect browsers' security in several ways, unrelated to encryption, protocols or the certificate used (these are tested by SSL Labs).
For more details about those headers, I'd suggest you navigate to securityheaders.io itself, and then check your site in it, then follow the links for each suggestion. Meanwhile, if your site uses Nginx, here's a "quick and dirty" guide for getting an A rating (and, yes, I'll mention what you'd need for an A+ rating, and also why I choose not to try for one).
Forces TLS (don't use if you still need to provide HTTP); prevents mixed content warnings. NOTE: again, this is not the most secure setting; see below.
You're probably wondering about the last two settings. I choose to send referring information to other sites for a simple reason: I like to see where my own visitors come from (I don't sell, share or monetize that in any way, it's just for curiosity's sake), and I'm a firm believer in treating others as I want to be treated. If you don't care about that, you may want to change this to "no-referrer-when-downgrade", or "strict-origin".
As for the Content Security Policy, anything more "secure" than the above prevents (or at least makes it a lot more headache-inducing) the use of inline scripts and/or external scripts, which would mean no external tools/scripts/content (or at least a lot of work in adding every external domain to a list of exceptions), and a lot of hacking on software such as WordPress (seriously: add the setting, but then remove the "unsafe-*" options and see what stops working...). You might use the most paranoid settings for an internally developed, mostly static site, but I fail to see the point. So, this and the paragraph above are the reasons for an A rating, instead of an A+.
I’m more of a systems administrator than a network admin (though I’ve also worked as the latter, in the past), but, of course, one can’t be a sysadmin without knowing at least something about networking. And yet (and like my previous post about compiling stuff), I’ve found that it’s possible to do a perfectly good job as a sysadmin, and yet still mix up a few networking concepts from time to time.
Therefore, I wanted to write about four different, but related, concepts, that I’ve noticed people sometimes confuse.
1. Allowing traffic
This just means that a network interface allows traffic to a specific destination and/or port, possibly restricted to a specific source (an IP address, or a network). Any traffic that is not allowed is simply refused.
Note that this by itself doesn’t mean that the interface (of a server, a router, etc.) will do anything (or anything desirable, at least) with the received traffic. For instance, it may not have something listening on that port, or it may not be configured to route that traffic to somewhere useful. This just means that it doesn’t instantly block the connection.
This is typically controlled by a local software firewall such as iptables.
2. Port forwarding
Port forwarding just means: if you receive a packet on interface A, port B, then redirect it to IP address C, port D. “D” may be the same as “B”, and “C” may be on the same host or at the other end of the planet.
Again, note that the mere fact that there’s a matching redirect rule for the initial destination interface and port doesn’t mean that the host actually accepts redirecting the traffic (see 1.) or that it knows how to route it to its final destination (see 4.)
3. NAT
NAT, or Network Address Translation, can be seen as a special case of two-way port forwarding (see 2.). Basically, a router (which may well be a simple server with two or more (physical or virtual) network interfaces, it doesn’t have to be a “router” bought in a store) accepts traffic from a (typically private) network, then translates it so that it goes to the destination address in the (typically public) network, with (and this is the important part) the source “masked” as the router’s public IP address, and a different source port that the router “remembers”, and then knows how to handle the returning traffic and “untranslate” it so that it goes to the original source.
In short, NAT allows many private hosts in a network to access the Internet using a single public IP, and a single connection. (It can have other similar uses, even some not related to the public Internet, of course; this is just the most common one.)
4. Routing
Routing is simply a host knowing that a packet intended for IP address X should be directed to IP address Y 1.
Again, the obligatory caveats: this doesn’t mean that the traffic is even accepted before attempting to route it (see 1.), or that the host actually knows how to reach IP address Y itself (it may not have a local route to it). It may also be that the routing is correct, but there is no address translation (see 3.), so the destination host receives a packet in a public interface that claims to be from a private address, and refuses it. Finally, the destination host itself needs a route to the original source, and it may not have one configured (or have a misconfigured one, causing asynchronous routing and possibly making the source refuse the returning traffic).
Thoughts? Corrections? Clarifications? Yes, I know this is relatively basic stuff. 🙂
MySQL and MariaDB(note: for brevity, I’ll be referring to “MySQL” from now on, but everything here applies to both and, indeed, these days I use MariaDB exclusively on my own servers) are very popular as database servers, but in my experience are rarely properly configured to take advantage of the server’s resources. While I’m not a database administrator (DBA), as a sysadmin I often have to diagnose performance problems in servers I maintain, and MySQL is a common culprit; it seems that most MySQL DBAs are well-versed in SQL, but the my.cnf configuration file seems to be a mostly unexplored mystery to them: either it’s untouched from the default configuration (which is conservatively set to work on the barest of machines), or they limit the memory usage to some 3% of the server’s available RAM(!) 1, or else they go in the other direction and try to use more resources than are available.
Without, of course, attempting to be a full MySQL guide (or even a full optimization guide), this tutorial will provide a few guidelines, rules-of-thumb, and settings suggestions so that a MySQL non-expert can still make his/her server run a lot more smoothly.
A vital tool:
MySQLTuner is an essential tool for MySQL administration, and, indeed, you should run it on your MySQL system before any configuration changes (whether suggested here or not), and also after implementing them. Its results, however, can be a bit intimidating, and (as its documentation warns) its recommendations should never be implemented blindly 2.
So, *do* install and use MySQLTuner (and, by following this guide, you should be seeing a lot more “green” and less “red” in its results), but don’t blindly follow its suggestions without understanding their point.
Oh, and keep your MySQLTuner script mostly up-to-date (assuming you installed it from its site, instead of your distro’s packaging system); don’t just keep using the version you downloaded some 4 years ago. Software evolves, you know. 🙂
About memory:
Apart from other “always-a-good-idea” settings that we’ll see later, the amount of memory you set MySQL to use is probably the most important configuration.
Before we begin: I still keep seeing co-workers (both at my job and the previous one) thinking of free memory as something that is desirable for a production server to have a lot of. They see a machine with, say, 90% of its RAM in use and get worried.
Guys and girls, Linux is not MS-DOS! (Really, you don’t have to obsess about freeing conventional memory anymore! 🙂 ) There’s such a thing as virtual memory, and also another thing called disk caching! Linux actually uses most of its free memory to cache disk access (and recent versions of top actually show that as still “available”, but for decades it said “cached”). Memory that’s truly free is simply being wasted (and a server with a large amount of it even after a week of uptime is a server that should have some of its RAM removed to be better used elsewhere).
Having more than half of a machine’s RAM as disk cache is certainly not as bad as having it unused/wasted, but it would have been far better to let applications (in this case, MySQL) actually, you know, use it. Which we’ll do here.
OK, end of rant. 🙂
So, what’s a good rule of thumb for how much memory to give to MySQL? I’d say 80% of the system’s unused RAM.
Note the “unused”. If the server just has MySQL on it, then, sure, give MySQL 80% of the free memory after loading up the OS. But if it runs other services (e.g. a web server, some other database, etc.), then see how much RAM you have after all of them are running, and give MySQL 80% of that.
An exception to this rule might be a very, very small database, say, with a single table with just a couple of entries, and rarely or never updated. In this case, giving it so much memory might be overkill… but in such a case, you wouldn’t be here reading an optimization guide, right?
Memory configuration:
(Note: there are other memory-related settings, such as join_buffer_size or innodb_log_buffer_size, but their defaults should be good enough. Also note that defaults may change (they’re usually increased) between MySQL/MariaDB versions, which is one more reason to keep your database server updated as much as possible.)
(Note 2: if you’re using very old versions of MySQL or MariaDB, it’s possible that some settings below don’t “exist” yet. If the server refuses to (re)start and complains about an unknown setting, simply remove it/comment it out. Better yet, start planning a software update…)
First, back up your database. Really. Nothing here should be dangerous, but better safe than sorry. And, if possible, try out these changes on a test machine before moving to the real, production one.
For extra fun, run MySQLTuner now and save the results (e.g. mysqltuner.pl > ~/mysql-pre-optimization.txt), so you can compare them later…
So, edit (after backing it up, of course) your /etc/my.cnf (or /etc/mysql/my.cnf in Ubuntu, or /etc/my.cnf.d/server.cnf in Fedora), and look for the [mysqld] section.
You can add the following entries to the end of that section, they’ll replace their settings if they’ve already been previously set.
If you’re using InnoDB (and, between it and MyISAM, you certainly should):
innodb_buffer_pool_size = MEM1
innodb_log_file_size = MEM2
innodb_buffer_pool_instances = NUMBER
MEM1 should be about 80% of your available RAM (see “About memory” above). You can specify it as a number followed by “M” for megabytes, “G” for gigabytes, etc.;
MEM2 should be one eighth (1/8) of MEM1. For example, for 1 GB of MEM1, set MEM2 to 128M;
NUMBER should be the number of gigabytes (rounded down) of MEM1. For instance, for 4 GB, set NUMBER to 4. Note that this value doesn’t have a unit (M, G, etc.)
If you’re not using InnoDB at all (why?), then don’t add any of the above, of course.
For MyISAM:
key_buffer_size = MEM3
If your databases are all InnoDB, then leave this value small (e.g. 4M), as the MySQL user tables and such are still MyISAM. If you have many and/or large MyISAM tables, on the other hand, set MEM3 to 25% of the available RAM (see “About memory” above), and reduce the InnoDB memory by that amount (see MEM1 above). If you have only MyISAM tables, then you can raise MEM3 to 50%-80% of available memory.
If you’re using Aria tables, the setting is aria_pagecache_buffer_size; follow the key_buffer_size recommendations.
Other settings:
I find the following settings to work pretty well in most cases (and MySQLTuner seems to agree):
query_cache_type = 0 # recommended to be off, these days
query_cache_size = 0
thread_cache_size = 128
table_open_cache = 2048
low_priority_updates = 1 # MyISAM only, but no harm
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
tmp_table_size = 64M
max_heap_table_size = 64M
And that’s it! Restart MySQL, and you should enjoy much better performance, both for MySQL itself and for the rest of the server.
As with most other internet services, Dovecot can be configured to use TLS encryption — and, unlike some others (such as web servers or SMTP servers), there’s little reason not to enforce it.
Notes:
Like other recent TLS tutorials on this blog, this is, of course, not a full Dovecot guide — that would be far too complex for a blog post. Instead, it’s just focused on enabling/configuring TLS encryption. I’ll be assuming you already have Dovecot configured, and are able to access mailboxes on your server using an email client.
I’ll be focusing on IMAP only, not POP3 or anything else.
The server will be configured to require encryption, for both privacy and security reasons. I don’t know of any modern email client that doesn’t allow encrypted connections (and even then, you might work around it by configuring an encrypted tunnel, but that falls out of this post’s scope).
1. Getting a certificate
While I believe most modern email clients will support ECDSA certificates, such clients are not a known quantity like, say, web browsers are, so I suggest that you create an RSA certificate, unless you already have one whose Common Name (CN) matches your server’s public name (e.g. mail.domain.com). Dovecot versions 2.2.31 and newer support configuring alternative certificates so that you could support both kinds at the same time, but there’s probably little gain in doing that, since most clients will likely default to RSA anyway (and, unless you’re using Let’s Encrypt, certificates are not free).
So, put your certificate and private key in /etc/dovecot/. Let’s assume the certificate is called myserver-full.crt1, and the private key is myserver.key. You should protect the private key from any users on your server, so do, for instance,
I’m assuming your Dovecot installation has a basic configuration file in /etc/dovecot/dovecot.conf, but the real “meat” of the configuration is in included files in /etc/dovecot/conf.d/. Your system may be slightly different, but I’m sure you can adapt. 🙂 Also, the numbers at the beginning of the file names may differ in your system.
This one isn’t really related to TLS, but it’s a good idea: edit /etc/dovecot/conf.d/20-imap.conf, and look for a line like this:
mail_plugins = $mail_plugins
and add to it (separated by a space):
imap_zlib
After all, there’s no reason not to use compression here, and bandwidth (especially on mobile) is still precious.
Now, edit /etc/dovecot/conf.d/10-ssl.conf, and add 2:
(The ssl_cipher_list line, besides setting secure defaults, sets the ChaCha20 protocol as the first one to be tried, since it’s considered one of the fastest and most secure. Note that it’ll require Dovecot linked to LibreSSL or OpenSSL 1.1.x to use that cipher, though Dovecot won’t complain if it doesn’t have access to it; it’ll just use the normal, secure defaults.)
If you don’t change anything else from the default configuration, the server will be listening on port 143 (IMAP) and port 993 (IMAPS). “But,” you ask, “isn’t standard IMAP unencrypted? I thought you were enforcing encryption…” Yes, but the standard IMAP port can be “upgraded” to TLS by entering the STARTTLS command, and email clients not only support that, but typically default to it (if not, just make sure you enable it). The server will refuse to authenticate any users on port 143 before they’ve “STARTTLSed”.
If you wanted to use only IMAP+STARTTLS, or only IMAPS, just edit /etc/dovecot/conf.d/10-master.conf, look for the “service” configuration, and disable the one you don’t want. But I see no problem with keeping both enabled.
Although Postfix (and the SMTP protocol in general) can function without any kind of encryption, enabling TLS it can be a good idea in terms of both security and privacy, so let’s look at how it can be easily done.
We’ll actually be configuring two separate types of encryption:
Opportunistic encryption for regular SMTP (port 25), both incoming1 and outgoing2. “Opportunistic”, here, means that the server will ask for encryption and use it if the other side also supports it, but if it doesn’t then it’ll work without encryption. Although forcing it might sound tempting for security reasons, in reality many “smaller” servers around the world still don’t support it, so you would be unable to send or receive mail to/from those. Opportunistic TLS at least means that most “big” email services (e.g. Gmail) will communicate with you (and you with them) with encryption.
Mandatory encryption for the submission service (port 587). This is used by your own users (even if it’s just you) to send mail through your server, and typically has different restrictions (e.g. it’s authenticated, but on the other hand it can be used to send mail to the outside (unlike incoming port 25 which doesn’t, as that would make it an open relay)). Both the users’ authentication that would otherwise be in clear text, and the fact that only a limited number of users (your users, too) will be accessing that port through email clients (never other email servers that you don’t control) make forcing encryption here a no-brainer.
NOTE: naturally, this is not an exhaustive Postfix tutorial. I’ll be assuming you already have a working server, and just want to add TLS encryption to it. Also, some parts may not apply to your case.
1. Have/get a TLS certificate for your email host’s public name
If you don’t already have one, see how to create a new TLS certificate. We’ll be using an RSA certificate here (instead of going with ECDSA) since unlike, say, a web server accessed by standard browsers, you can’t control what other email servers support 3, therefore it’s best to choose the most common option. 4
Note that the certificate’s CN must match your email host’s public name (e.g. “mail.domain.com“). That will also probably correspond to (one of) the domain’s MX records.
2. Postfix configuration
Again, I’ll be assuming your non-TLS Postfix is already working fine.
Put your certificate and key in /etc/postfix (for instance). For this example, I’ll be calling them myserver-full.crt and myserver.key.
In /etc/postfix/main.cf, add the following lines:
# TLS configuration starts here
tls_random_source = dev:/dev/urandom
# openssl_path=/usr/local/libressl/bin/openssl
# uncomment and edit the above if you're using a different "openssl" than the system's
# (in this case, LibreSSL)
# SMTP from your server to others
smtp_tls_key_file = /etc/postfix/myserver.key
smtp_tls_cert_file = /etc/postfix/myserver-full.crt
smtp_tls_CAfile = /etc/postfix/myserver-full.crt
smtp_tls_security_level = may
smtp_tls_note_starttls_offer = yes
smtp_tls_mandatory_protocols=!SSLv2,!SSLv3
smtp_tls_protocols=!SSLv2,!SSLv3
smtp_tls_loglevel = 1
smtp_tls_session_cache_database =
btree:/var/lib/postfix/smtp_tls_session_cache
# SMTP from other servers to yours
smtpd_tls_key_file = /etc/postfix/myserver.key
smtpd_tls_cert_file = /etc/postfix/myserver-full.crt
smtpd_tls_CAfile = /etc/postfix/myserver-full.crt
smtpd_tls_security_level = may
smtpd_tls_auth_only = yes
smtpd_tls_mandatory_protocols=!SSLv2,!SSLv3
smtpd_tls_protocols=!SSLv2,!SSLv3
smtpd_tls_loglevel = 1
smtpd_tls_session_cache_database =
btree:/var/lib/postfix/smtpd_tls_session_cache
# TLS configuration ends here
And in /etc/postfix/master.cf, assuming you already have a “submission” section a bit like this:
submission inet n - n - - smtpd
-o smtpd_etrn_restrictions=reject
-o smtpd_sasl_auth_enable=yes
-o smtpd_recipient_restrictions=permit_mynetworks,permit_sasl_authenticated,reject
-o smtpd_client_restrictions=permit_mynetworks,permit_sasl_authenticated,reject
-o smtpd_helo_restrictions=permit_mynetworks,permit
Add to that section:
-o smtpd_tls_security_level=encrypt
The above forces encryption for the submission service (remember that it’s not required for normal SMTP, it’s just desired).
In short:
your server tries to connect to others using TLS, but falls back to an unencrypted connection if the other side doesn’t support encryption;
other servers can connect to yours using TLS, but if they don’t attempt it then the connection will be unencrypted;
your users *must* use encryption (and authentication) to send mail through your server.
Restart your server, and check the logs: you should be getting mentions of TLS now. For instance, an email server starting an encrypted connection to yours looks like:
Sep 6 14:25:58 drax postfix/smtpd[22727]: Anonymous TLS connection established from lists.openbsd.org[192.43.244.163]: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
Most people who maintain web servers (or email servers, or…) have probably had to deal with SSL/TLS certificates in the past, but the process is, in my opinion, typically more complex than it should be, and not that well documented, typically forcing the user to consult several tutorials on the web so that they can adapt parts of each for their needs. Since I had to renew a couple of certificates last week, I thought about writing a quick and easy tutorial for a particular case: HTTPS on Nginx (although the certificate itself could, of course, be used for other services; right now I have one I’m using for IMAP (Dovecot) and SMTP (Postfix) as well). For extra fun, I’ll be creating/using an ECDSA certificate with an EC key, instead of the more usual RSA type; ECDSA is more modern and is theoretically more secure even with smaller keys.
A few notes:
There are alternatives (e.g. key sizes, etc.) for basically every parameter I’m using, but I’m not going into those. This is supposed to be as quick and easy as possible, after all.
Similarly, I’m not going into Let’s Encrypt; instead, I’m assuming a “normal” certificate authority such as Comodo (which supports ECDSA certificates; if you choose another, you should make sure of that in advance). If you do use Let’s Encrypt, just use it to create the certificate instead of the “send the CSR to the certification authority” part.
The certificate (and server) will be compatible with most browsers, but that “most” won’t include any Microsoft browsers on Windows XP (Firefox or Chrome on that abomination of an OS will still work).
If you already have a certificate (whether ECDSA or not, whether from Let’s Encrypt or not) and you’re here just for the A+ rating on SSL Labs, skip to “Setting up Nginx” below.
Generating the key and the Certificate Signing Request (CSR):
The second command will ask you for details about your server/company (location, etc.). You should fill in every field, although the only mandatory one is “Common Name” (CN), which must match your server’s public name (not necessarily the machine’s name, but the host name people will type in the browser, such as “zurgl.com“. Note that a certificate for “domain.com” also includes “www.domain.com” (so don’t include the “www.” in the CN), but if the server is reached at “subdomain.domain.com“, then that’s what the CN needs to be.
Ordering and receiving the new certificate:
Now go to a certification authority (CA), order a new certificate, and when asked for a CSR, send them (usually you can just copy and paste it to a text entry window) that myserver.csr file.
If everything went well, then the CA should email you the new certificate in a short while. Typically they send you two files: the certificate itself, and a couple of “intermediate” certificates. Only the first is really needed, but I’ve had best results with concatenating your certificate (first) and the intermediate certs (last) into a single file, which you might call myserver-full.crt . Put it somewhere Nginx can access (e.g. /etc/nginx), and also the key you generated earlier (in this example, myserver.key). You don’t need the CSR there, by the way; it was just needed for ordering the new certificate.
Setting up Nginx:
This is, of course, not a complete Nginx tutorial (that would take a lot more space than a single post), just a simple recipe for configuring an HTTPS site with your new certificate, and have it be secure (and get a great score at SSL Labs, too). I’m assuming you can take care of all the non-HTTPS bits.
So, inside a virtual host, you need the server section:
server {
listen 443 ssl http2; # if your nginx complains about 'http2', remove it, or (better yet) upgrade to a recent version
server_name mysite.com # replace with your site, obviously
# all the non-HTTPS bits (directories, log paths, etc.) go here
# the rest of the recipe -- see below -- can go here
}
See the last comment? OK, let’s begin adding stuff there (I won’t indent any configuration lines from now on, but they look better if aligned with the rest of the configuration inside the { } ).
ssl_certificate /etc/nginx/myserver-full.crt; # the certificate and the intermediate certs, as seen above...
ssl_certificate_key /etc/nginx/myserver.key; # ... and the key
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS:!3DES';
# the above comes from Mozilla's Server Side TLS guide
add_header X-Content-Type-Options nosniff;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 5m; # basic defaults.
Great, you now have a basic HTTPS server! I suggest you try it now: save the configuration, restart Nginx, and test it. Access the URL with a browser (and check if the browser doesn’t complain about the certificate: if it did, something went wrong), and run it through SSL Labs’ Server Test. Check if it complains about something; if so, something needs fixing (ask in the comments, maybe I or someone else can help).
If all went well, you probably got a decent score, maybe even an A. But how to get an A+, like I promised at the beginning?
An A+ score on SSL Labs’ Server Test:
As of now (September 2017), SSL Labs only asks for one more thing: “HTTP Strict Transport Security (HSTS) with long duration”. HSTS is a mechanism to tell browsers: “this site should be accessed through HTTPS only; if you attempt to connect through normal HTTP, then deny access”. This information is actually cached by the browser, it’s not the server that refuses HTTP connections (though it’s, of course, possible to configure Nginx to do just that — or simply do 301 redirects to the equivalent HTTPS URL. But I digress…). In short, it protects against downgrade attacks. As for the “with long duration” part, that’s simply the time that browsers should cache that information.
IMPORTANT: don’t proceed until you have the basic HTTPS site running, and SSL Labs reporting no problems (see the previous section)!
ALSO IMPORTANT: if you *do* want your site to be accessible through both HTTP and HTTPS, stop here, as the rest of the configuration will make it HTTPS-only! That means, however, giving up on the A+ score.
Restart Nginx, try SSL Labs’ test again. Did you get an A+?
If not, or you have any questions, feel free to ask.
For extra fun: have your Nginx use LibreSSL or OpenSSL 1.1.x, and enjoy a few more modern ciphers (look for “ChaCha20” on SSL Labs) without any configuration changes from the above.
As with most other internet services,