How to get a 100% score on SSL Labs (Red Hat/CentOS 7.x, Apache, Let’s Encrypt)

There’s an Nginx version of this tutorial already, but since some people prefer/have to use Apache, here’s how to get 100% on SSL Labs’ server test with it.

For extra fun, instead of Debian/Ubuntu, we’ll be using Red Hat/CentOS (more precisely, CentOS 7.6 with the latest updates as of January 10th, 2019, but this shouldn’t change for any other 7.x versions of either CentOS or RHCE). And we’ll keep firewalld and SELinux enabled 1, even though, as far as I know, in most companies (well, at least the ones I’ve worked at) it’s, typically, official policy to disable both. 🙂

So, let’s assume you already have a CentOS/RHEL machine, with internet access and the default repositories enabled. First, we’ll enable the EPEL repository (to provide Let’s Encrypt’s certbot):

yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

Now we’ll install certbot (which will also install Apache’s httpd and mod_ssl as dependencies) and wget (which we may need later):

yum install python2-certbot-apache wget

To be run certbot and allow it to verify your site, you should first create a simple vhost. Add this to your Apache configuration (e.g. a new file ending in .conf in /etc/httpd/conf.d/):

<VirtualHost *:80>
    DocumentRoot "/var/www/html"
    ServerName mysite.mydomain.com
</VirtualHost>

Now add Apache’s ports to firewalld, and start the service:

firewall-cmd --permanent --zone=public --add-service=http
firewall-cmd --permanent --zone=public --add-service=https
firewall-cmd --reload

systemctl enable httpd ; systemctl restart httpd

Run certbot to create and begin using the new certificate:

certbot --apache --rsa-key-size 4096 --no-redirect --staple-ocsp -d mysite.mydomain.com

You should now have the virtual host configured in /etc/httpd/conf.d/ssl.conf . Let’s add a few lines to it (inside the <VirtualHost></VirtualHost> section):

Header always set Strict-Transport-Security "max-age=15768000"
SSLProtocol             all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite          ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384::ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384
SSLCipherSuite          TLSv1.3 TLS_CHACHA20_POLY1305_SHA256:TLS_AES_256_GCM_SHA384
SSLHonorCipherOrder     on
SSLCompression          off

(Don’t mind the “TLSv1.3” line for now, it won’t be used — if at all  — until later in this post, and won’t have any effect if it isn’t.)

This should get you an A+ rating, with all bars at 100% except for Key Exchange. A good start, right? 🙂 However, that final bar can be a bit tricky. The problem here is that SSL Labs limits the Key Exchange rating to 90% if the default ECDH curve is either prime256v1 or X25519, the first of them being the OpenSSL default… and Apache 2.4.6 (the version included in all 7.x RHELs/CentOSes) doesn’t yet include the option to specify a different list of curves (of which secp384r1 will give a 100% rating, and won’t add any compatibility issues).

Assuming you don’t want to install a non-official Apache, then the only way (as far as I know) to specify a new curve is to add the results of running:

openssl ecparam -name secp384r1

to your certificate file. But Let’s Encrypt certificates are renewed often, so you may need some trickery here — such as adding something like this to root’s crontab:

*/5 * * * * grep -q "EC PARAMETERERS" /your/certificate/file.crt || openssl ecparam -name secp384r1 >> /your/certificate/file.crt

Ugly, I know. 🙂

Alternatively, you can upgrade Apache to a newer version. CodeIT (for instance) provides replacement packages for RHEL and CentOS, which you can use by doing:

cd /etc/yum.repos.d && wget https://repo.codeit.guru/codeit.el`rpm -q --qf "%{VERSION}" $(rpm -q --whatprovides redhat-release)`.repo
yum -y update httpd mod_ssl

This will update Apache from 2.4.6 to 2.4.37 (as of January 10th, 2019), and so you can add the following to the vhost:

SSLOpenSSLConfCmd     Curves secp384r1:X25519:prime256v1

However! This will still not give 100%, since CoreIT’s Apache appears to come with a newer OpenSSL (1.1.1) statically linked to it, which means it’ll enable TLS 1.3 by default. And there’s nothing wrong with it… except that, from all the tests I’ve performed, Apache ignores the specified order of the ECDH curves when using TLS 1.3. With the line above, it’ll use secp384r1 for TLS 1.2, but X25519 for TLS 1.3 — and that’s enough to lower the Key Exchange bar to 90% again. 🙁

Given this, there are now several options:

1- disable TLS 1.3 (it’s still relatively new, and most people are still using 1.2): change the line:

SSLProtocol             all -SSLv3 -TLSv1 -TLSv1.1

to:

SSLProtocol             all -SSLv3 -TLSv1 -TLSv1.1 -TLSv1.3

2- force secp384r1: change the “SSLOpenSSLConfCmd Curves” line to simply:

SSLOpenSSLConfCmd       Curves secp384r1

3- ignore this situation altogether: X25519 is considered to be as secure as secp384r1, if not more; even SSL Labs mentions not penalizing it in the future in their (early 2018) grading guide — they just have been a bit lazy in updating their test. It’s quite possible that a future update of their tool will give a full 100% score to X25519.

Anyway, most of this was done for fun, for the challenge of it.  🙂

Red Hat/CentOS: How to add HTTPS to an existing Nginx website (both with and without Let’s Encrypt)

(Yes, you read the title correctly. For extra fun, and to prevent this blog from being too focused on Ubuntu/Debian, this time I’ll be using Red Hat Enterprise Linux / CentOS (and, I assume, Fedora as well.) Later on, I may post a Debian-based version.)

Configuring a basic HTTP site on Nginx

(Note: if you already have a working HTTP site, you can skip to the next section (“Adding encryption…”))

Yes, the post title mentions an “existing” website, which I believe will be the case in most “real world” situations, but installing a new one is actually very easy on CentOS1. First, do:

yum -y install epel-release; yum -y install nginx

Then create a very basic configuration file for the (non-HTTPS) site, as /etc/nginx/conf.d/mysite.conf :

server {
        listen 80;
        server_name mysite.mydomain.com;
        root /var/www/mysite;

        location / {
        }
}

Then, of course, create the /var/www/mysite directory (CentOS doesn’t use /var/www by default, but I’m far too used to it to change. 🙂 ) If you’d like, create an index.html text file in that directory, restart nginx (“service nginx restart” or “systemctl restart nginx“, depending on your system’s major version), and try browsing to http://mysite.mydomain.com . If it works, congratulations, you have a running web server and a basic site.

Adding encryption to the site (not using Let’s Encrypt):

First, you need a key/certificate pair. I have tutorials for creating an RSA certificate or an ECDSA certificate.

Second, edit the site’s configuration file (in the “starting from scratch” example above, it’s “/etc/nginx/conf.d/mysite.conf“), and copy the entire server section so that it appears twice on that text file (one after the other). Pick either the original or the copy (not both!), and, inside it, change the line:

listen 80;

to:

listen 443 ssl http2;

(Note: the “http2” option is only available in Nginx 1.9.5 or newer. If your version complains about it, just remove it, or upgrade.)

Inside that section, add the following options:

ssl_certificate /path/to/certificate.crt;
ssl_certificate_key /path/to/privatekey.crt;

(replacing the paths and file names, of course.)

This should be enough — restart Nginx and you should have an HTTPS site as well as the HTTP one.

And what if you want to disable HTTP for that site and use HTTPS only? Just edit the same configuration file, look for the server section you didn’t change (the one that still includes “listen 80;“), and replace the inside of that section with:

listen 80;
server_name mysite.domain.com;
return 301 https://mysite.domain.com$request_uri;

Afterward, while you’re at it, why not go for an A+ rating on SSL Labs (skip the certification creation part from that tutorial, you’ve done it already) and an A rating on securityheaders.io?

Adding encryption to the site (using Let’s Encrypt):

First, install certbot:

yum install python2-certbot-nginx

Then check if Nginx is running and your normal HTTP site is online (it’ll be needed in a short while, to activate the certificate). If so, then enter:

certbot --nginx --rsa-key-size 4096 -d mysite.mydomain.com

(replacing “mysite.mydomain.com” with yours, of course.)

Answer the questions it asks you: a contact email, whether you agree with the terms (you need to say yes to this one), if you want to share your email with the EFF, and finally if you want “No redirect” (i.e. keep the HTTP site) or “Redirect” (make your site HTTPS only).

And that’s it (almost — see the next paragraph) — when you get the shell prompt back, certbot will already have reconfigured Nginx in the way you chose in the paragraph above, and restarted it so that it’s running the new configuration. You may want to add “http2” to the “listen 443 ssl;” line in the configuration file (it’ll probably be the default someday, but as of this post’s date it isn’t), and don’t forget your options for improved security and security headers.

Only one thing is missing: automatically renewing certificates. Strangely, the certbot package configures that automatically on Ubuntu, but not on CentOS, from what I’ve seen (please correct me if I’m wrong). The official Let’s Encrypt docs recommend adding this (which includes some randomization so that entire timezones don’t attempt to renew their certificates at precisely the same time) to root’s crontab:

0 0,12 * * * python -c 'import random; import time; time.sleep(random.random() * 3600)' && certbot renew 

(Note: It’s possible to use Let’s Encrypt to create ECDSA certificates, but as of this writing you have to do most of the work manually (creating a CSR, etc.), and you lose the automatic renewal, so for the moment I suggest using RSA certificates. I hope this changes in the future.)

Linux: Creating and using an encrypted data partition

Encrypted disks and/or filesystems are nothing new on Linux; most distributions, these days, allow you to encrypt some or all your disk partitions, so that they can only be accessed with a password. In this tutorial, however, we’re going to add a new encrypted partition to an existing system, using only the command line. I’ve found that tutorials on the web seem to make this issue more complex than it actually is, so here’s mine — hopefully it’ll be easier than most.

In this guide, we’ll be making a few assumptions. First, as said above, it’ll be an existing system, to which we’ve just added a new disk, /dev/sdb . Adapt to your situation, of course. If you’re not using an entire disk, just create a new partition with fdisk (e.g. /dev/sdb1) and use it instead. Second, we’ll have the machine boot with the disk unmounted, then use a command to mount it (asking for a passphrase, of course), and another to dismount it when you don’t need it. The obvious usage of such a partition is for storing sensitive/private data, but theoretically, you could run software on it — as long as you don’t automatically attempt to start it on boot, and don’t mind keeping it mounted most or all of the time, which perhaps defeats its purpose: if it’s mounted, it’s accessible.

So, without further ado…

Continue reading “Linux: Creating and using an encrypted data partition”

Linux: How to Compile Stuff

A couple of days ago I was surprised when someone who’s worked as a Linux sysadmin for several years asked me how to compile a program that was only available in source format (it was a cryptocurrency miner, for the curious), since he’d never compiled anything in his life. How was that possible? But then I realized: compiling stuff on Linux hasn’t really been a necessity 1 for the last, oh, 15-20 years or so. Back in 1993 or so (yikes!), when I started using Linux, things were of course quite different.

So, without further ado: how to compile stuff on Linux. Assuming you’ve already downloaded and uncompressed a program’s source, here’s what works 99% of the time (from the program’s directory, of course):

./configure --help  # to see available options

./configure # followed by the desired options, if any

make && make install

By default, compiled programs are then installed to /usr/local (e.g. binaries go to /usr/local/bin, libraries to /usr/local/lib , etc.). If you want to change that, just add –prefix=path (e.g. –prefix=/usr/local/test) to the ./configure options. (In that example, it’ll install binaries to /usr/local/test/bin, and so on. There are ways (i.e. different ./configure options) to install to different sub-paths, but those are beyond this basic tutorial. The default /usr/local/bin, lib, etc. paths also have the advantage of being in the default executable and library paths in typical distributions, so that you don’t have to add to those.)

If the above commands worked for you, congratulations, you’ve compiled your first program, which should now be ready to run! 🙂 However, if you’ve never compiled anything before on that system, it’s more likely that the ./configure command complained about missing libraries, missing utilities, or even a missing compiler; after all, as the first paragraph showed, it’s possible to use Linux for years, even work with it for a living, without needing to compile anything, which in turn has caused most modern distributions to not install compilers and development stuff by default.

How to do so? On Debian/Ubuntu, start with:

apt-get install build-essential

And on Red Hat/CentOS/Fedora, use:

yum groupinstall “Development Tools”

The above should install the C/C++ compiler and common development tools. However, a particular program may ask for other libraries’ development files, which may not be installed even though the library itself is. It’s impossible to give you an exhaustive list, but typically you can use yum search or apt-cache search to find what you need. Debian-based development packages typically end in -dev, while Red Hat-based ones end in -devel . For instance, suppose you’re on Ubuntu and ./configure complains about not having the zlib development files. A quick search would point you (at least on Ubuntu 17.04) to zlib1g-dev, which you’d install with apt-get install zlib1g-dev .

And what if there is no configure script (unlikely these days, but possible if it’s a very old source)? If there is a file named Makefile, then try typing make, and see if it compiles. Otherwise, your best bet is to read any provided documentation (e.g. a README or INSTALL file).

Any questions, feel free to ask.

Red Hat / CentOS: How to list RPMs installed on year X

Paranoid-minded auditor: “I need a list of all RPMs that were installed this year on these servers, stat!”

You: “OK, let’s see here…”

On a single machine:

for i in `rpm -qa`; do rpm -qi $i | grep Install | grep -q 2017 && echo $i; done

If you need to check several servers, you can use a bash “for” cycle, or pssh, or…

How to update Red Hat or CentOS without changing minor versions

If you’re a Linux system administrator or even a “mere” user, you’ve probably noticed that, when using a Red Hat-like system, if you do “yum update” it may well raise the minor version level (e.g. 6.7 to 6.9). In fact, it should move your system to the latest minor version (the number after the dot) of your current major version (the number before it).

You may, then, have wondered if it is possible to update your system and yet remain on your current version. You may even have been asked to do so by a very, very timid boss, or some development/application team (“this is supported only on Red Hat 7.1, we can’t move to 7.2!”).

Before I go on, I have to say that there is absolutely no technical reason to do this (EDIT: not necessarily true any longer, at least for 7.x, see CertDepot’s comment and link. Still true in most cases; the reason for this demand is almost always ignorance, fear, and laziness, not knowledge of any actual change causing incompatibilities). I really hope you’ve arrived here just because a boss, project manager or developer is demanding it (and, sadly, you don’t work at a place where you can say “no, that’s stupid, I won’t do it”… yet 😉 ), or simply because of scientific curiosity, not because you actually think that doing this is a good idea.

Red Hat (or CentOS) minor versions aren’t really” versions” in the usual sense, where new versions of software packages, libraries, etc. are included. Instead, (with a few desktop-related exceptions, such as web browsers) they take pains to only fix security problems and other bugs. If you look at a particular package’s versions, whether you’re on Red Hat Enterprise Linux 7.0 or 7.3, those always stay the same, only the “Red Hat” number (e.g. file-5.11-21.el7) increases. Therefore, there is never (EDIT: see above edit) any question of “compatibility”; it may, however, be a question of “officially supported”, which is code for “we tested our product with this version, and can’t be bothered to test it with any others.”

Sorry about the rant. 🙂 So, since you’re obviously a competent sysadmin, I’ll assume you’re being forced to do it. Here’s how:

With Satellite:

To see which releases you have available:

subscription-manager release --list

Example:

# subscription-manager release --list
+-------------------------------------------+
 Available Releases
+-------------------------------------------+
5.11
5Server
6.2
6.7
6.8
6.9
6Server
7.0
7.1
7.2
7.3
7Server

To lock on a release (e.g. 7.1):

subscription-manager release --set=7.1

And to unlock it:

subscription-manager release --unset

(or maybe –set=7Server)

Without Satellite:

For a single update, add –releasever=x.y to your yum command; for instance:

yum --releasever=7.1 update

To set it permanently, add:

distroverpkg=x.y

to the [main] section in your /etc/yum.conf file.

Notes: at least on CentOS, since CentOS 7.x, versions aren’t just “x.y”, they also include a third number, apparently the year and date of release. Browsing on http://vault.centos.org/centos/ , for instance, you see you have these versions available:

[DIR] 6.7/ 21-Jan-2016 13:22 - 
[DIR] 6.8/ 24-May-2016 17:36 - 
[DIR] 6.9/ 10-Apr-2017 12:48 - 
[DIR] 6/ 10-Apr-2017 12:48 - 
[DIR] 7.0.1406/ 07-Apr-2015 14:36 - 
[DIR] 7.1.1503/ 13-Nov-2015 13:01 - 
[DIR] 7.2.1511/ 18-May-2016 16:48 - 
[DIR] 7.3.1611/ 20-Feb-2017 22:23 - 
[DIR] 7/ 20-Feb-2017 22:23 -

and, yes, you have to specify the third number in your command/config file.

You may also have to enable the several entries in your /etc/yum.repos.d/CentOS-Vault.repo file (change enabled=0 to 1).

Sources: 1