How to get a 100% score on SSL Labs (Nginx, Let’s Encrypt)

This post should be titled “How to get a 100% score on SSL Labs (Nginx, Let’s Encrypt) as of April 2018“, since SSL Labs‘s test evolves all the time (and a good thing it does, too.) But that would be too long a title. 🙂

The challenge:

As we’ve seen before, it’s relatively easy to get an A+ rating on SSL Labs. However, even that configuration will only give a score like this:

Zurgl.com's current SSL Labs score
Zurgl.com’s current SSL Labs score

Certainly more than good enough (and, incidentally, almost certainly much better than whatever home banking site you use…), but your weirdo of a host can’t look at those less-than-100% bars and not see a challenge. 🙂

First, note that all of the following will be just for fun, as the settings to get 100% will demand recent browsers (and, in some cases, recent operating systems/devices), so you probably don’t want this in a world-accessible site. It would be more applicable, say, for a webmail used by you alone, or by half a dozen people. But the point here isn’t real-world use, it’s fun (OK, OK, and learning something new). So, let’s begin.

Getting that perfect SSL Labs score:

For this example, I’ll be using a just-installed Debian 9 server, in which I did something like:

apt -y install nginx
echo "deb http://ftp.debian.org/debian stretch-backports main" >> /etc/apt/sources.list.d/stretch-backports.list # if you don't have this repository active already
apt -y install python-certbot-nginx -t stretch-backports

(The reason for the above extra complexity is the fact that the default certbot in Debian Stretch is too old and tries to authenticate new certificates in an obsolete way, but there’s a newer one in stretch-backports.)

I also created a simple index.html file on /var/www/html/ that just says “Hello world!”. Right now, the site doesn’t even have HTTPS.

So, let’s create a new certificate and configure the HTTPS server in Nginx:

certbot --nginx --rsa-key-size 4096 --no-redirect --staple-ocsp -d mysite.mydomain.com

(replacing mysite.mydomain.com, obviously.)

Testing it on SSL Labs, it gives… exactly the same bars as in the image above, but with an A rating instead of A+. Right, we need HSTS to get A+ (and currently certbot doesn’t support configuring it automatically in Nginx), so we add this to the virtual host’s server section:

add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";

(Note: remove “includeSubDomains;” if you have HTTP sites on any subdomain. Also, HSTS makes your site HTTPS-only, so you can’t use it if you want to keep an HTTP version — in which case you can’t get better than an A rating.)

This upgrades the rating to A+, as expected, but the bars still didn’t change. Let’s make them grow, shall we?

Certificate:

It’s already at 100%, so yay. 🙂

Protocol Support:

Just disable any TLS lower than 1.2. In this case, edit /etc/letsencrypt/options-ssl-nginx.conf and replace the ssl_protocols line (or comment it and add a new one) with:

ssl_protocols TLSv1.2;

WARNING: any change you make to the file above will affect all virtual hosts on this Nginx server. If you need some of them to support older protocols, it’s better to just comment out that option in that file, and then include it in each virtual host’s server section.

EDIT: actually, it turns out that ssl_protocols affects the entire server, with the first option Nginx finds  (first in /etc/nginx.conf, then in the default website (which possibly includes /etc/letsencrypt/options-ssl-nginx.conf)) taking precedence. So just choose whether you want TLS 1.0 to 1.2 (maximum compatibility) or just 1.2 only (maximum security/rating); if you need both, use two separate servers.

Key Exchange:

We’ve already created the new certificate with a 4096-bit key (with “rsa-key-size 4096“, see above), and according to SSL Labs’ docs this should be enough… but it isn’t. After some googling, I found out that you also need to add the following option inside the server section in Nginx:

ssl_ecdh_curve secp384r1;

Since the default Nginx+OpenSSL/LibreSSL setting, either “X25519” or “secp256r1” (actually “prime256v1“), also lowers the score.

EDIT: again, ssl_ecdh_curve affects the entire server, so you can’t use different default curves for each virtual host. So I’d suggest (in /etc/letsencrypt/options-ssl-nginx.conf):

ssl_ecdh_curve secp384r1:X25519:prime256v1;

if you want the 100% rating, or:

ssl_ecdh_curve X25519:prime256v1:secp384r1:

otherwise. This one doesn’t affect compatibility, by the way; it’s just a question of the preferred order.

The certificate’s key size (4096 or 2048) is, like the certificate itself, specific to each virtual host.

Cipher Strength:

For 100% here, you need to disable not only any old protocols, but also any 128-bit ciphers. I started from Mozilla’s SSL tool (with the “Modern” option selected), then removed anything with “128” in its name, then moved ChaCha20 to the front just because, and ended up with this line, which I added to /etc/letsencrypt/options-ssl-nginx.conf (replacing the current setting with the same name):

ssl_ciphers 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384::ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384';

(This time, this setting can indeed be specified for each virtual host, though you’d need to comment it out in the file above (which affects all virtual hosts with Let’s Encrypt certificates) if you want to do so, otherwise they’ll conflict.)

The result:

100% on SSL Labs

As said near the beginning, this is not something you likely want to do (yet) for a production, world-accessible site, as it will require relatively modern browsers, and you typically can’t control what visitors use1. According to SSL Labs’ list, it’s not actually that bad: all versions of Chrome, Firefox, or Safari from the past couple of years are fine, but no Microsoft browser older than IE11 will work, nor will Android’s default browser before 7.0 (Chrome on Android, which is not the same thing, will do fine.) I’d suggest it for a site where you can “control” the users, such as a small or medium-sized company’s webmail or employee portal (where you, as a sysadmin, can and should demand up-to-date security from your users).

But the main point of this exercise was, of course, to see if I could do it. Challenging yourself is always good. 🙂 And if you can share what you learned with others, so much the better.

Red Hat/CentOS: How to add HTTPS to an existing Nginx website (both with and without Let’s Encrypt)

(Yes, you read the title correctly. For extra fun, and to prevent this blog from being too focused on Ubuntu/Debian, this time I’ll be using Red Hat Enterprise Linux / CentOS (and, I assume, Fedora as well.) Later on, I may post a Debian-based version.)

Configuring a basic HTTP site on Nginx

(Note: if you already have a working HTTP site, you can skip to the next section (“Adding encryption…”))

Yes, the post title mentions an “existing” website, which I believe will be the case in most “real world” situations, but installing a new one is actually very easy on CentOS1. First, do:

yum -y install epel-release; yum -y install nginx

Then create a very basic configuration file for the (non-HTTPS) site, as /etc/nginx/conf.d/mysite.conf :

server {
        listen 80;
        server_name mysite.mydomain.com;
        root /var/www/mysite;

        location / {
        }
}

Then, of course, create the /var/www/mysite directory (CentOS doesn’t use /var/www by default, but I’m far too used to it to change. 🙂 ) If you’d like, create an index.html text file in that directory, restart nginx (“service nginx restart” or “systemctl restart nginx“, depending on your system’s major version), and try browsing to http://mysite.mydomain.com . If it works, congratulations, you have a running web server and a basic site.

Adding encryption to the site (not using Let’s Encrypt):

First, you need a key/certificate pair. I have tutorials for creating an RSA certificate or an ECDSA certificate.

Second, edit the site’s configuration file (in the “starting from scratch” example above, it’s “/etc/nginx/conf.d/mysite.conf“), and copy the entire server section so that it appears twice on that text file (one after the other). Pick either the original or the copy (not both!), and, inside it, change the line:

listen 80;

to:

listen 443 ssl http2;

(Note: the “http2” option is only available in Nginx 1.9.5 or newer. If your version complains about it, just remove it, or upgrade.)

Inside that section, add the following options:

ssl_certificate /path/to/certificate.crt;
ssl_certificate_key /path/to/privatekey.crt;

(replacing the paths and file names, of course.)

This should be enough — restart Nginx and you should have an HTTPS site as well as the HTTP one.

And what if you want to disable HTTP for that site and use HTTPS only? Just edit the same configuration file, look for the server section you didn’t change (the one that still includes “listen 80;“), and replace the inside of that section with:

listen 80;
server_name mysite.domain.com;
return 301 https://mysite.domain.com$request_uri;

Afterward, while you’re at it, why not go for an A+ rating on SSL Labs (skip the certification creation part from that tutorial, you’ve done it already) and an A rating on securityheaders.io?

Adding encryption to the site (using Let’s Encrypt):

First, install certbot:

yum install python2-certbot-nginx

Then check if Nginx is running and your normal HTTP site is online (it’ll be needed in a short while, to activate the certificate). If so, then enter:

certbot --nginx --rsa-key-size 4096 -d mysite.mydomain.com

(replacing “mysite.mydomain.com” with yours, of course.)

Answer the questions it asks you: a contact email, whether you agree with the terms (you need to say yes to this one), if you want to share your email with the EFF, and finally if you want “No redirect” (i.e. keep the HTTP site) or “Redirect” (make your site HTTPS only).

And that’s it (almost — see the next paragraph) — when you get the shell prompt back, certbot will already have reconfigured Nginx in the way you chose in the paragraph above, and restarted it so that it’s running the new configuration. You may want to add “http2” to the “listen 443 ssl;” line in the configuration file (it’ll probably be the default someday, but as of this post’s date it isn’t), and don’t forget your options for improved security and security headers.

Only one thing is missing: automatically renewing certificates. Strangely, the certbot package configures that automatically on Ubuntu, but not on CentOS, from what I’ve seen (please correct me if I’m wrong). The official Let’s Encrypt docs recommend adding this (which includes some randomization so that entire timezones don’t attempt to renew their certificates at precisely the same time) to root’s crontab:

0 0,12 * * * python -c 'import random; import time; time.sleep(random.random() * 3600)' && certbot renew 

(Note: It’s possible to use Let’s Encrypt to create ECDSA certificates, but as of this writing you have to do most of the work manually (creating a CSR, etc.), and you lose the automatic renewal, so for the moment I suggest using RSA certificates. I hope this changes in the future.)

How to configure SPF in Postfix

SPF, or Sender Policy Framework, is a way for domain owners to say: “these are the email servers my domain sends mail from; anyone else is attempting to impersonate me.” These days, a lot of email services implement it in some way (both for receiving and sending email), and, if you administer your own server, it’s quite easy to implement, as we’ll see.

There are actually two parts to SPF: configuring it for your domain (so that your outgoing emails have a better chance of being accepted and emails impersonating your domain have a better chance of being refused), and configuring your email server to check whether incoming emails come from an SPF authorized server or not. Without further ado:

1- Configuring SPF for your domain’s outgoing mail:

This is actually done in DNS, not in the email server. Simply add a TXT record for your domain, with something like this:

v=spf1 a:mailserver1.domain.com a:mailserver2.domain.com -all

There are other configuration parameters, such as getting the servers from the domain’s MX record (add “mx“), and copying another domain’s authorized servers (“include:otherdomain.com“). And, of course, if you only have one server sending mail, you don’t need two “a:” entries.

Pay special attention to the “-all” part at the end: that instructs other servers to refuse any email apparently coming from your domain that isn’t authorized for it. That should ideally be the final configuration, but if you’re doing this for the first time you may want to change it to “~all“, which means “softfail”: it suggests to the other side that the mail is probably not legit, but that it shouldn’t refuse it outright because of it. After testing it for a while, if you see no problems then you can change it to “-all“.

2- Configuring SPF checking for incoming mail in Postfix:

I’ll assume you already have a working Postfix server (hopefully already using TLS encryption, hint hint), and that you want to respect any “-all” configurations (see above) in incoming mail — in other words, if a domain says “all my mail comes from server X, refuse anything else” and you receive an email purportedly from that domain, but from a server Y, it’s refused (for “~all“, it’d generate a warning).

First, install pypolicyd-spf. On a Debian-based system, the package (as of this writing) still has its old name, so enter the following:

apt install postfix-policyd-spf-python

Edit /etc/postfix/master.cf, and add:

# spf
policyd-spf  unix  -       n       n       -       0       spawn
        user=policyd-spf argv=/usr/bin/policyd-spf

Then edit /etc/postfix/main.cf, and look for the “smtpd_recipient_restrictions” section. That section will probably begin by accepting mail from your networks and from authenticated users, then rejecting non-authorized relaying, possibly followed by some white- or blacklists, and maybe a couple of filters, finally ending with “permit“. Before that “permit“, add:

    check_policy_service unix:private/policyd-spf,

(note the comma at the end; without it, Postfix wouldn’t read the rest of that section.)

IMPORTANT: while you’re still in testing, it may be a good idea to add “warn_if_reject” to the beginning of the previous configuration line. With it, Postfix will still log SPF failures, but won’t actually reject an email because of it. When you’re certain everything is fine, you can then remove that prefix (and then reload Postfix, of course).

Add also the following to that configuration file, outside of any section:

policy-spf_time_limit = 3600

And you’re done. Reload or restart Postfix, and check your logs to see if everything is working as intended.

Note that other tools, such as SpamAssassin, may also use SPF to weigh whether an incoming mail is more or less likely to be spam. That, however, is unrelated to the Postfix configuration above, which does a binary “block/don’t block”.

Nginx: How to prevent it from logging local hits

Like any web server, Nginx logs all accesses/hits by default, and if you have some kind of log-based analytics tool (such as AWstats) you probably have it already set to ignore hits from the host itself (usually by skipping both localhost (127.0.0.1) and the server’s public IP address). If you don’t do so, then your statistics will probably be inflated (“wow, my newly created site is surprisingly popular!“), and, unless your site has many regular users, the IP address at the top of the “visitors” table will almost certainly be your own. Local hits can come from several culprits: for instance, some web software such as WordPress or MyBB use special URLs as a form of cron replacement, and also you may have yourself be monitoring your site in some way (such as, in my case, benchmarking a particular URL and MRTGing the access speed).

OK, so your analytics don’t show it anymore — but what about the log files themselves? Maybe you don’t want a large percentage of them being composed of internal hits (especially in the case of those “let’s measure the average response speed for 100 hits every 5 minutes” benchmarks, which for that site made up some 95% of its access logs…). Maybe it even interferes with some other tools you’re using, such as something like DenyHosts or Fail2ban, to detect some abuse patterns, even after whitelisting your external IP. How about having the option not to log them at all? 1

On Nginx, this is actually pretty easy to do.

1- in the Nginx main configuration file, add the following:

map $remote_addr $notlocal {
        default 1;
        ~^YOUR_EXTERNAL_IP$ 0;
        ~^127.0.0.1$ 0;
}

Replacing YOUR_EXTERNAL_IP with… you can probably guess. 🙂

Important: if your virtual hosts’ configurations are in another directory that is “included” from nginx.conf, take care to add the above before the include lines (e.g. include /etc/nginx/conf.d/*.conf) . Otherwise, the virtual hosts won’t have the “$notlocal” variable defined yet, and Nginx won’t start because of it.

2- for each virtual host where you want to stop logging local hits, edit the access_log line, changing it from something like:

access_log /var/log/nginx/mysite-access.log;

to:

access_log /var/log/nginx/mysite-access.log combined if=$notlocal;

Note the addition of “combined“. “combined” is the default format for Nginx access logs, so specifying it isn’t usually needed, but apparently if you want to specify the “if” condition, it must come after the log format option (otherwise, I guess Nginx thinks you want a log format of “if“, which of course doesn’t exist and will prevent the server from starting.)

That it! Restart Nginx, and enjoy your much cleaner/smaller logs.

Stuff you can do quickly with MRTG (that has nothing to do with router traffic)

Come on, everyone knows MRTG, right? That venerable tool for graphing router traffic, among other things. If you’ve worked as a sysadmin and/or network admin in the past, you’re probably familiar with it since decades ago, although these days you probably use something more modern, such as Cacti, Zabbix, etc..

The thing is, I still use MRTG a lot, old though it is, and even though there are many newer alternatives, since MRTG has, to me, one huge advantage. Well, two, in fact, though they really amount to one thing: ease of use. MRTG is simple: it doesn’t need to run as a daemon, it’s just a binary you can run through cron. And MRTG is versatile: unlike other tools that really, really want to work with network traffic, or system memory, or processor usage, or something else (preferably to be read through SNMP, or a local agent), MRTG just wants one or two values, and it’ll call the first one “input” and the second “output”. And those values can come from anywhere.

Just to show how easy it is: an MRTG configuration file usually looks something like this:

WorkDir:/var/www/htdocs/ptmail/

Refresh: 36000

Target[ptmail]: `/usr/local/bin/getptmail.sh`;
Options[ptmail]: growright, nopercent, integer, noinfo, gauge
MaxBytes[ptmail]: 12500000000000
kilo[ptmail]: 1000
YLegend[ptmail]: MB used / Emails
ShortLegend[ptmail]:  
LegendI[ptmail]:  MB used (uncompressed):
LegendO[ptmail]:  Emails:
Title[ptmail]: MB used / Emails
PageTop[ptmail]: <h1>MB used / Emails</h1>

XSize[ptmail]: 500
YSize[ptmail]: 250
XScale[ptmail]: 1.4
YScale[ptmail]: 1.4

Timezone[ptmail]: Europe/Lisbon

The key part is that “Target” line which, you’ll notice, is between backticks, meaning that it’ll actually execute a script. And what does it expect from that script? Simple: just four lines:

  • A number (which it’ll think of as “input”, and which will typically be graphed in green)
  • Another number (for “output”, to be graphed in blue)
  • The system’s uptime (on Linux, uptime | cut -d ” ” -f 4-7 | cut -d “,” -f 1-2 typically provides it in the format MRTG wants, although it’s really just text)
  • The system’s name (just use uname -n)

That’s it! If you only want to graph a single value, just add “noi” or “noo” to the Options line (and you can then replace the first or second line, respectively, with “echo 0” on your script). You can also make it not show the uptime and name with the “noinfo” option. And, finally, the “gauge” option makes MRTG graph current values, while, without it, it adds the current value to the previous one (like, for instance, a “bytes transferred” counter). Both have their uses.

Now, you’re probably thinking: “yeah, yeah, I’ve known how to use MRTG for decades (and these days I use XYZ instead; who even uses MRTG in 2017?!?); why are you writing about something so basic and, well, old?” The answer is, again, 1) that it’s ridiculously easy to use (just create a script to write 4 lines), and 2) that it’s not for routers (or network interfaces) only; it can be used for so much else, and it can typically be done in minutes. So I’ll just show a few examples of stuff I already do with it on my servers. Each example will be just the shell script; the MRTG configuration file for these is virtually always the same thing, except for axis legends, labels, etc..

Note: all of the following need the “gauge” option.

“Steal” time on a VPS

#!/bin/bash

# Graphs "steal" time on a VPS. A high value means you need to complain to your
# VPS provider that the host your server is on is overbooked, or there's
# another customer abusing it (mining Bitcoins? 🙂 )

NUM=$((3 + ($RANDOM % 3)))

rm -f /tmp/steals.txt

top -b -n $NUM | grep Cpu | cut -d ',' -f 8 | tr -d ' ' | cut -d 's' -f 1 > /tmp/steals.txt

TOTAL=0 ; for i in `cat /tmp/steals.txt`; do TOTAL=`echo "$TOTAL + $i" | bc -l`; done

AVERAGE=`echo "$TOTAL / $NUM" | bc -l`

echo 0
echo $AVERAGE
uptime | cut -d " " -f 4-7 | cut -d "," -f 1-2
uname -n

Total size and number of emails in a Maildir

#!/bin/bash

# Total size and number of mails in a Maildir

# total UNCOMPRESSED size of your mailbox, in MB, 2 decimal cases
# actual compressed size will probably be much lower; if you'd rather show it,
# just replace the following with a "du -ms /home/ptmail/Maildir | cut -f 1"
find /home/ptmail/Maildir -type f | grep drax | cut -d '=' -f 2 | cut -d ',' -f 1 > /tmp/sizes-ptmail ; perl -e "printf \"%.2f\n\", `paste -s -d+ /tmp/sizes-ptmail | bc` / 1024 / 1024"

# number of emails
find /home/ptmail/Maildir -type f | grep drax | wc -l

uptmailime | cut -d " " -f 4-7 | cut -d "," -f 1-2
uname -n

System load (* 100) and number of processes

#!/usr/bin/php
<?php

# Graph system load (multiplied by 100) and number of processes.

# Yes, this is trivial and uses mostly shell commands. 🙂 It's just to show
# that "scripts" invoked by MRTG can be in any language (in this case, PHP),
# not just shell scripts. You could even use compiled C code, for instance.

$x = exec ("uptime | cut -d ':' -s -f 5 | cut -d ',' -f 1 | cut -d ' ' -f 2");

$x*=100;

print $x . "\n";

system ("ps ax | wc -l");

system ("uptime | cut -d ' ' -f 4-7 | cut -d ',' -f 1-2");
system ("uname -n");

?>

Average time per request of a URL

#!/bin/bash

# Average time per request of the URL below. Run it locally to measure how fast
# your site and/or server is, or remotely to measure its network connection as well
# (though that would be less reliable; best use something like Pingdom)

# use "noi" for this one (hence the first "echo 0")

echo 0
ab -c 4 -n 100 -k https://zurgl.com/ | grep "Time per request" | head -n 1 | cut -d ':' -f 2 | tr -d ' ' | cut -d '[' -f 1

uptime | cut -d " " -f 4-7 | cut -d "," -f 1-2
uname -n

I could go on, but I’m sure you get the idea. 🙂