Zurgl

March 24, 2018April 9, 2018

Nginx: How to prevent it from logging local hits

Like any web server, Nginx logs all accesses/hits by default, and if you have some kind of log-based analytics tool (such as AWstats) you probably have it already set to ignore hits from the host itself (usually by skipping both localhost (127.0.0.1) and the server’s public IP address). If you don’t do so, then your statistics will probably be inflated (“wow, my newly created site is surprisingly popular!“), and, unless your site has many regular users, the IP address at the top of the “visitors” table will almost certainly be your own. Local hits can come from several culprits: for instance, some web software such as WordPress or MyBB use special URLs as a form of cron replacement, and also you may have yourself be monitoring your site in some way (such as, in my case, benchmarking a particular URL and MRTGing the access speed).

OK, so your analytics don’t show it anymore — but what about the log files themselves? Maybe you don’t want a large percentage of them being composed of internal hits (especially in the case of those “let’s measure the average response speed for 100 hits every 5 minutes” benchmarks, which for that site made up some 95% of its access logs…). Maybe it even interferes with some other tools you’re using, such as something like DenyHosts or Fail2ban, to detect some abuse patterns, even after whitelisting your external IP. How about having the option not to log them at all? ¹

On Nginx, this is actually pretty easy to do.

1- in the Nginx main configuration file, add the following:

map $remote_addr $notlocal {
        default 1;
        ~^YOUR_EXTERNAL_IP$ 0;
        ~^127.0.0.1$ 0;
}

Replacing YOUR_EXTERNAL_IP with… you can probably guess. 🙂

Important: if your virtual hosts’ configurations are in another directory that is “included” from nginx.conf, take care to add the above before the include lines (e.g. include /etc/nginx/conf.d/*.conf) . Otherwise, the virtual hosts won’t have the “$notlocal” variable defined yet, and Nginx won’t start because of it.

2- for each virtual host where you want to stop logging local hits, edit the access_log line, changing it from something like:

access_log /var/log/nginx/mysite-access.log;

to:

access_log /var/log/nginx/mysite-access.log combined if=$notlocal;

Note the addition of “combined“. “combined” is the default format for Nginx access logs, so specifying it isn’t usually needed, but apparently if you want to specify the “if” condition, it must come after the log format option (otherwise, I guess Nginx thinks you want a log format of “if“, which of course doesn’t exist and will prevent the server from starting.)

That it! Restart Nginx, and enjoy your much cleaner/smaller logs.

March 21, 2018April 9, 2018

Linux: What’s filling up my almost full filesystem?

Let’s say you realize (maybe because you got an alarm for it) that a particular filesystem — let’s say /qwerty — is full, or almost full, and you want to find out what’s taking up the most space. Simply enter something like:

du -k -x /qwerty | sort -n | tail -20

to get a list of the directories taking up the most space in that filesystem, sorted by size, with the largest ones at the bottom.

Note that the “/qwerty“, in this case, should be the filesystem’s top directory, not some subdirectory of it. In other words, it should be something that shows up on a “df” command.

Explanation:

“du -k -x” shows the subdirectories (and their total sizes) of the specified directory (or of the current one, if you don’t specify one). “-k” means that sizes are reported in kilobytes (KB) — this is not mandatory, but different versions of “du” may use other units, and this one is easy to read. “-x” means “don’t go out of the specified filesystem”, and that’s quite important here, as you could have a “/qwerty/asdfgh” filesystem, mounted in a subdirectory of “/qwerty“, but since it’s a different filesystem (appearing separately in the results of a “df” command) you won’t want to include it in this list.

“sort -n” is a simple numeric sort, and “tail -20” just means “show only the last 20 lines. 20 is a reasonable number that fits in most terminals at their default sizes (typically 80×25), so that you don’t have to scroll up.

(Yes, this is very basic, but, hey, one of the goals of this blog is to document stuff I help co-workers with, since, at least in theory, if someone has a question about something they need to do, then many other people may have the same doubt/need as well, and we have a couple of new team members whose backgrounds are not Linux/Unix-related, so…)

March 15, 2018April 9, 2018

Linux: Creating and using an encrypted data partition

Encrypted disks and/or filesystems are nothing new on Linux; most distributions, these days, allow you to encrypt some or all your disk partitions, so that they can only be accessed with a password. In this tutorial, however, we’re going to add a new encrypted partition to an existing system, using only the command line. I’ve found that tutorials on the web seem to make this issue more complex than it actually is, so here’s mine — hopefully it’ll be easier than most.

In this guide, we’ll be making a few assumptions. First, as said above, it’ll be an existing system, to which we’ve just added a new disk, /dev/sdb . Adapt to your situation, of course. If you’re not using an entire disk, just create a new partition with fdisk (e.g. /dev/sdb1) and use it instead. Second, we’ll have the machine boot with the disk unmounted, then use a command to mount it (asking for a passphrase, of course), and another to dismount it when you don’t need it. The obvious usage of such a partition is for storing sensitive/private data, but theoretically, you could run software on it — as long as you don’t automatically attempt to start it on boot, and don’t mind keeping it mounted most or all of the time, which perhaps defeats its purpose: if it’s mounted, it’s accessible.

So, without further ado…

Continue reading “Linux: Creating and using an encrypted data partition”

March 11, 2018April 9, 2018

Nginx: How to get an A (or A+) rating on securityheaders.io

Some time ago, I wrote about how to set up Nginx to get an A+ rating on SSL Labs (focusing specifically on an ECDSA certificate, but you can certainly achieve the same with a (more common) RSA certificate). SSL Labs' test is very comprehensive, warning the user of most potential problems, including SSL/TLS implementation bugs (update your software!), misconfigurations, problems with the certificate itself, etc.. But there is another similar tool, securityheaders.io, which focuses on something different: security-related headers. Such headers are sent by the web server (such as Nginx or Apache) and affect browsers' security in several ways, unrelated to encryption, protocols or the certificate used (these are tested by SSL Labs).

For more details about those headers, I'd suggest you navigate to securityheaders.io itself, and then check your site in it, then follow the links for each suggestion. Meanwhile, if your site uses Nginx, here's a "quick and dirty" guide for getting an A rating (and, yes, I'll mention what you'd need for an A+ rating, and also why I choose not to try for one).

Nginx setting	Explanation
add_header Strict-Transport-Security "max-age=31536000; includeSubdomains; preload";	Enforces HTTPS on the entire site. Don't use if you still need to provide HTTP, of course.
add_header X-Content-Type-Options nosniff;	Prevents browsers from trying to "guess" MIME types and such, forcing them to use what the server tells them.
add_header X-Frame-Options SAMEORIGIN;	Stops your site from being included in iframes on other sites.
add_header X-Xss-Protection "1" always;	Activates cross-scripting (XSS) protection in browsers.
add_header Referrer-Policy "unsafe-url";	Makes the site always send referrer information to other sites. NOTE: this is not the most secure setting, but it's the one I prefer; see below.
add_header Content-Security-Policy "default-src https: data: 'unsafe-inline' 'unsafe-eval'";	Forces TLS (don't use if you still need to provide HTTP); prevents mixed content warnings. NOTE: again, this is not the most secure setting; see below.

You're probably wondering about the last two settings. I choose to send referring information to other sites for a simple reason: I like to see where my own visitors come from (I don't sell, share or monetize that in any way, it's just for curiosity's sake), and I'm a firm believer in treating others as I want to be treated. If you don't care about that, you may want to change this to "no-referrer-when-downgrade", or "strict-origin".

As for the Content Security Policy, anything more "secure" than the above prevents (or at least makes it a lot more headache-inducing) the use of inline scripts and/or external scripts, which would mean no external tools/scripts/content (or at least a lot of work in adding every external domain to a list of exceptions), and a lot of hacking on software such as WordPress (seriously: add the setting, but then remove the "unsafe-*" options and see what stops working...). You might use the most paranoid settings for an internally developed, mostly static site, but I fail to see the point. So, this and the paragraph above are the reasons for an A rating, instead of an A+.

September 25, 2017April 9, 2018

Networking definitions: Allowing traffic, Port forwarding, NAT, and Routing

I’m more of a systems administrator than a network admin (though I’ve also worked as the latter, in the past), but, of course, one can’t be a sysadmin without knowing at least something about networking. And yet (and like my previous post about compiling stuff), I’ve found that it’s possible to do a perfectly good job as a sysadmin, and yet still mix up a few networking concepts from time to time.

Therefore, I wanted to write about four different, but related, concepts, that I’ve noticed people sometimes confuse.

1. Allowing traffic

This just means that a network interface allows traffic to a specific destination and/or port, possibly restricted to a specific source (an IP address, or a network). Any traffic that is not allowed is simply refused.

Note that this by itself doesn’t mean that the interface (of a server, a router, etc.) will do anything (or anything desirable, at least) with the received traffic. For instance, it may not have something listening on that port, or it may not be configured to route that traffic to somewhere useful. This just means that it doesn’t instantly block the connection.

This is typically controlled by a local software firewall such as iptables.

2. Port forwarding

Port forwarding just means: if you receive a packet on interface A, port B, then redirect it to IP address C, port D. “D” may be the same as “B”, and “C” may be on the same host or at the other end of the planet.

Again, note that the mere fact that there’s a matching redirect rule for the initial destination interface and port doesn’t mean that the host actually accepts redirecting the traffic (see 1.) or that it knows how to route it to its final destination (see 4.)

3. NAT

NAT, or Network Address Translation, can be seen as a special case of two-way port forwarding (see 2.). Basically, a router (which may well be a simple server with two or more (physical or virtual) network interfaces, it doesn’t have to be a “router” bought in a store) accepts traffic from a (typically private) network, then translates it so that it goes to the destination address in the (typically public) network, with (and this is the important part) the source “masked” as the router’s public IP address, and a different source port that the router “remembers”, and then knows how to handle the returning traffic and “untranslate” it so that it goes to the original source.

In short, NAT allows many private hosts in a network to access the Internet using a single public IP, and a single connection. (It can have other similar uses, even some not related to the public Internet, of course; this is just the most common one.)

4. Routing

Routing is simply a host knowing that a packet intended for IP address X should be directed to IP address Y ¹.

Again, the obligatory caveats: this doesn’t mean that the traffic is even accepted before attempting to route it (see 1.), or that the host actually knows how to reach IP address Y itself (it may not have a local route to it). It may also be that the routing is correct, but there is no address translation (see 3.), so the destination host receives a packet in a public interface that claims to be from a private address, and refuses it. Finally, the destination host itself needs a route to the original source, and it may not have one configured (or have a misconfigured one, causing asynchronous routing and possibly making the source refuse the returning traffic).

Thoughts? Corrections? Clarifications? Yes, I know this is relatively basic stuff. 🙂

September 17, 2017March 19, 2019

Stuff you can do quickly with MRTG (that has nothing to do with router traffic)

Come on, everyone knows MRTG, right? That venerable tool for graphing router traffic, among other things. If you’ve worked as a sysadmin and/or network admin in the past, you’re probably familiar with it since decades ago, although these days you probably use something more modern, such as Cacti, Zabbix, etc..

The thing is, I still use MRTG a lot, old though it is, and even though there are many newer alternatives, since MRTG has, to me, one huge advantage. Well, two, in fact, though they really amount to one thing: ease of use. MRTG is simple: it doesn’t need to run as a daemon, it’s just a binary you can run through cron. And MRTG is versatile: unlike other tools that really, really want to work with network traffic, or system memory, or processor usage, or something else (preferably to be read through SNMP, or a local agent), MRTG just wants one or two values, and it’ll call the first one “input” and the second “output”. And those values can come from anywhere.

Just to show how easy it is: an MRTG configuration file usually looks something like this:

WorkDir:/var/www/htdocs/ptmail/

Refresh: 36000

Target[ptmail]: `/usr/local/bin/getptmail.sh`;
Options[ptmail]: growright, nopercent, integer, noinfo, gauge
MaxBytes[ptmail]: 12500000000000
kilo[ptmail]: 1000
YLegend[ptmail]: MB used / Emails
ShortLegend[ptmail]:  
LegendI[ptmail]:  MB used (uncompressed):
LegendO[ptmail]:  Emails:
Title[ptmail]: MB used / Emails
PageTop[ptmail]: <h1>MB used / Emails</h1>

XSize[ptmail]: 500
YSize[ptmail]: 250
XScale[ptmail]: 1.4
YScale[ptmail]: 1.4

Timezone[ptmail]: Europe/Lisbon

The key part is that “Target” line which, you’ll notice, is between backticks, meaning that it’ll actually execute a script. And what does it expect from that script? Simple: just four lines:

A number (which it’ll think of as “input”, and which will typically be graphed in green)
Another number (for “output”, to be graphed in blue)
The system’s uptime (on Linux, uptime | cut -d ” ” -f 4-7 | cut -d “,” -f 1-2 typically provides it in the format MRTG wants, although it’s really just text)
The system’s name (just use uname -n)

That’s it! If you only want to graph a single value, just add “noi” or “noo” to the Options line (and you can then replace the first or second line, respectively, with “echo 0” on your script). You can also make it not show the uptime and name with the “noinfo” option. And, finally, the “gauge” option makes MRTG graph current values, while, without it, it adds the current value to the previous one (like, for instance, a “bytes transferred” counter). Both have their uses.

Now, you’re probably thinking: “yeah, yeah, I’ve known how to use MRTG for decades (and these days I use XYZ instead; who even uses MRTG in 2017?!?); why are you writing about something so basic and, well, old?” The answer is, again, 1) that it’s ridiculously easy to use (just create a script to write 4 lines), and 2) that it’s not for routers (or network interfaces) only; it can be used for so much else, and it can typically be done in minutes. So I’ll just show a few examples of stuff I already do with it on my servers. Each example will be just the shell script; the MRTG configuration file for these is virtually always the same thing, except for axis legends, labels, etc..

Note: all of the following need the “gauge” option.

“Steal” time on a VPS

#!/bin/bash

# Graphs "steal" time on a VPS. A high value means you need to complain to your
# VPS provider that the host your server is on is overbooked, or there's
# another customer abusing it (mining Bitcoins? :) )

NUM=$((3 + ($RANDOM % 3)))

rm -f /tmp/steals.txt

top -b -n $NUM | grep Cpu | cut -d ',' -f 8 | tr -d ' ' | cut -d 's' -f 1 > /tmp/steals.txt

TOTAL=0 ; for i in `cat /tmp/steals.txt`; do TOTAL=`echo "$TOTAL + $i" | bc -l`; done

AVERAGE=`echo "$TOTAL / $NUM" | bc -l`

echo 0
echo $AVERAGE
uptime | cut -d " " -f 4-7 | cut -d "," -f 1-2
uname -n

Total size and number of emails in a Maildir

#!/bin/bash

# Total size and number of mails in a Maildir

# total UNCOMPRESSED size of your mailbox, in MB, 2 decimal cases
# actual compressed size will probably be much lower; if you'd rather show it,
# just replace the following with a "du -ms /home/ptmail/Maildir | cut -f 1"
find /home/ptmail/Maildir -type f | grep drax | cut -d '=' -f 2 | cut -d ',' -f 1 > /tmp/sizes-ptmail ; perl -e "printf \"%.2f\n\", `paste -s -d+ /tmp/sizes-ptmail | bc` / 1024 / 1024"

# number of emails
find /home/ptmail/Maildir -type f | grep drax | wc -l

uptime | cut -d " " -f 4-7 | cut -d "," -f 1-2
uname -n

System load (* 100) and number of processes

#!/usr/bin/php
<?php

# Graph system load (multiplied by 100) and number of processes.

# Yes, this is trivial and uses mostly shell commands. :) It's just to show
# that "scripts" invoked by MRTG can be in any language (in this case, PHP),
# not just shell scripts. You could even use compiled C code, for instance.

$x = exec ("uptime | cut -d ':' -s -f 5 | cut -d ',' -f 1 | cut -d ' ' -f 2");

$x*=100;

print $x . "\n";

system ("ps ax | wc -l");

system ("uptime | cut -d ' ' -f 4-7 | cut -d ',' -f 1-2");
system ("uname -n");

?>

Average time per request of a URL

#!/bin/bash

# Average time per request of the URL below. Run it locally to measure how fast
# your site and/or server is, or remotely to measure its network connection as well
# (though that would be less reliable; best use something like Pingdom)

# use "noi" for this one (hence the first "echo 0")

echo 0
ab -c 4 -n 100 -k https://zurgl.com/ | grep "Time per request" | head -n 1 | cut -d ':' -f 2 | tr -d ' ' | cut -d '[' -f 1

uptime | cut -d " " -f 4-7 | cut -d "," -f 1-2
uname -n

I could go on, but I’m sure you get the idea. 🙂

September 11, 2017April 9, 2018

Linux: Detecting potential disk space problems before going home

As is typical in sysadmin work, one member of my team is always on call, having to be available during the night and on weekends. Sometimes the problems are big (especially hardware failures), sometimes they’re trivial, and sometimes they’re false alarms, but one thing is always true: it’s not fun to be woken up by a ringing phone, especially when you’re already not having enough sleep.

One of the most common reasons for being called at night (or during weekends) is when a disk’s occupation exceeds a threshold (let’s say 95%, although that value varies in reality). Logs get larger (especially when some error is constantly occurring), old logs aren’t compressed/rotated/deleted automatically, users leave huge debug files somewhere and forget to delete them for years… all of these happen, it’s mostly inevitable.

But, if we’re woken up at 4 A.M. because a /var partition reached 95% utilization, isn’t it true that in most cases the increase was gradual, and the value was already abnormally high — let’s say 93 or 94% — at 4 P.M.?

And wouldn’t it have been much better (for our well-earned rest, among other things) for the person on call to have fixed the problem during the day, before going home?

I thought so, too. Which is why I wrote a script some months ago to detect which servers (from more than a thousand) will need attention soon.

(Looking at that script now, I’m a bit ashamed to share it, as it’s pretty lazily coded, repeating code one time for each filesystem instead of having some kind of function… but it was made in a bit of a hurry some time ago, so because of laziness… I mean, historical accuracy 🙂 I’ll share it as it is (other than translating variable names and output messages from my native Portuguese)).

So, here it is. Don’t laugh too hard, OK? 🙂

#!/bin/bash

# check / partition
ROOT=`df -hl / | tail -1 | awk '{ print $5 }' | cut -d '%' -f 1`
case $ROOT in
    ''|*[!0-9]*) ROOT=`df -hl / | tail -1 | awk '{ print $4 }' | cut -d '%' -f 1` ;;
esac
case $ROOT in
    ''|*[!0-9]*) ROOT=`df -hl / | tail -1 | awk '{ print $3 }' | cut -d '%' -f 1` ;;
esac

TOP=$ROOT
WORST="/"

# check /var partition
VAR=`df -hl /var | tail -1 | awk '{ print $5 }' | cut -d '%' -f 1`
case $VAR in
    ''|*[!0-9]*) VAR=`df -hl /var | tail -1 | awk '{ print $4 }' | cut -d '%' -f 1` ;;
esac

if [ "$VAR" -gt "$TOP" ]; then
        TOP=$VAR
        WORST="/var"
fi

# check /home partition
# comment out this entire block if you don't want to monitor /home
HOME=`df -hl /home | tail -1 | awk '{ print $5 }' | cut -d '%' -f 1`
case $HOME in
    ''|*[!0-9]*) HOME=`df -hl /home | tail -1 | awk '{ print $4 }' | cut -d '%' -f 1` ;;
esac

if [ "$HOME" -gt "$TOP" ]; then
        TOP=$HOME
        WORST="/home"
fi


# check /tmp partition
TMP=`df -hl /tmp | tail -1 | awk '{ print $5 }' | cut -d '%' -f 1`
case $TMP in
    ''|*[!0-9]*) TMP=`df -hl /tmp | tail -1 | awk '{ print $4 }' | cut -d '%' -f 1` ;;
esac

if [ "$TMP" -gt "$TOP" ]; then
        TOP=$TMP
        WORST="/tmp"
fi


# check /opt partition
OPT=`df -hl /opt | tail -1 | awk '{ print $5 }' | cut -d '%' -f 1`
case $OPT in
    ''|*[!0-9]*) OPT=`df -hl /opt | tail -1 | awk '{ print $4 }' | cut -d '%' -f 1` ;;
esac

if [ "$OPT" -gt "$TOP" ]; then
        TOP=$OPT
        WORST="/opt"
fi


#echo "TOP = $TOP"

if [ "$TOP" -ge 70 ] && [ "$TOP" -le 80 ]; then
	echo "Worst partition: ($WORST) at $TOP"
        exit 7
fi


if [ "$TOP" -ge 80 ] && [ "$TOP" -le 90 ]; then
	echo "Worst partition: ($WORST) at $TOP"
        exit 8
fi

if [ "$TOP" -ge 90 ] && [ "$TOP" -le 95 ]; then
	echo "Worst partition: ($WORST) at $TOP"
        exit 9
fi

if [ "$TOP" -ge 95 ] && [ "$TOP" -lt 98 ]; then
	echo "Worst partition: ($WORST) at $TOP"
        exit 10
fi

if [ "$TOP" -eq 98 ]; then
	echo "Worst partition: ($WORST) at $TOP"
        exit 11
fi

if [ "$TOP" -eq 99 ]; then
	echo "Worst partition: ($WORST) at $TOP"
        exit 12
fi


if [ "$TOP" -gt 99 ]; then
	echo "Worst partition: ($WORST) at $TOP :("
        exit 20
fi

echo "Everything OK: worst partition: ($WORST) at $TOP"
exit 0

The goal is to run it on every server you’re responsible for (we use an HP automation tool where I work, but it could be done in many ways, including with a central machine who could ssh “passwordless-ly” into all other servers — it wouldn’t need root access for that, since it doesn’t change anything anywhere. Afterwards, you’re supposed to sort the output by exit codes; anything other than 0 might indicate a problem, and the higher, the worse it is; exit code 20 means a partition at 100%. Again, that HP tool does that sorting easily, but it’s far from being the only way — if you’re reading this, you’ve probably already thought of some.

And then, naturally, you’re supposed to go through the worst cases, either by deleting obvious trash, or by contacting the application teams so that they clean up their stuff.

Depending on your company’s policies, you may want to comment out the block that refers to the /home partition, if that is completely the responsibility of the users.

I don’t have statistics, of course, but I can tell you that being called because of a(n almost) full filesystem, which was relatively frequent a year or so ago, is now extremely rare (only happening when — after work hours — an application goes really wrong (typically because an untested settings change) and starts dumping error messages/debug logs as fast as it can). If only hardware problems didn’t exist, either…

September 9, 2017April 1, 2019

MySQL/MariaDB easy optimization tutorial

Introduction:

MySQL and MariaDB (note: for brevity, I’ll be referring to “MySQL” from now on, but everything here applies to both and, indeed, these days I use MariaDB exclusively on my own servers) are very popular as database servers, but in my experience are rarely properly configured to take advantage of the server’s resources. While I’m not a database administrator (DBA), as a sysadmin I often have to diagnose performance problems in servers I maintain, and MySQL is a common culprit; it seems that most MySQL DBAs are well-versed in SQL, but the my.cnf configuration file seems to be a mostly unexplored mystery to them: either it’s untouched from the default configuration (which is conservatively set to work on the barest of machines), or they limit the memory usage to some 3% of the server’s available RAM(!) ¹, or else they go in the other direction and try to use more resources than are available.

Without, of course, attempting to be a full MySQL guide (or even a full optimization guide), this tutorial will provide a few guidelines, rules-of-thumb, and settings suggestions so that a MySQL non-expert can still make his/her server run a lot more smoothly.

A vital tool:

MySQLTuner is an essential tool for MySQL administration, and, indeed, you should run it on your MySQL system before any configuration changes (whether suggested here or not), and also after implementing them. Its results, however, can be a bit intimidating, and (as its documentation warns) its recommendations should never be implemented blindly ².

So, *do* install and use MySQLTuner (and, by following this guide, you should be seeing a lot more “green” and less “red” in its results), but don’t blindly follow its suggestions without understanding their point.

Oh, and keep your MySQLTuner script mostly up-to-date (assuming you installed it from its site, instead of your distro’s packaging system); don’t just keep using the version you downloaded some 4 years ago. Software evolves, you know. 🙂

About memory:

Apart from other “always-a-good-idea” settings that we’ll see later, the amount of memory you set MySQL to use is probably the most important configuration.

Before we begin: I still keep seeing co-workers (both at my job and the previous one) thinking of free memory as something that is desirable for a production server to have a lot of. They see a machine with, say, 90% of its RAM in use and get worried.

Guys and girls, Linux is not MS-DOS! (Really, you don’t have to obsess about freeing conventional memory anymore! 🙂 ) There’s such a thing as virtual memory, and also another thing called disk caching! Linux actually uses most of its free memory to cache disk access (and recent versions of top actually show that as still “available”, but for decades it said “cached”). Memory that’s truly free is simply being wasted (and a server with a large amount of it even after a week of uptime is a server that should have some of its RAM removed to be better used elsewhere).

Having more than half of a machine’s RAM as disk cache is certainly not as bad as having it unused/wasted, but it would have been far better to let applications (in this case, MySQL) actually, you know, use it. Which we’ll do here.

OK, end of rant. 🙂

So, what’s a good rule of thumb for how much memory to give to MySQL? I’d say 80% of the system’s unused RAM.

Note the “unused”. If the server just has MySQL on it, then, sure, give MySQL 80% of the free memory after loading up the OS. But if it runs other services (e.g. a web server, some other database, etc.), then see how much RAM you have after all of them are running, and give MySQL 80% of that.

An exception to this rule might be a very, very small database, say, with a single table with just a couple of entries, and rarely or never updated. In this case, giving it so much memory might be overkill… but in such a case, you wouldn’t be here reading an optimization guide, right?

Memory configuration:

(Note: there are other memory-related settings, such as join_buffer_size or innodb_log_buffer_size, but their defaults should be good enough. Also note that defaults may change (they’re usually increased) between MySQL/MariaDB versions, which is one more reason to keep your database server updated as much as possible.)

(Note 2: if you’re using very old versions of MySQL or MariaDB, it’s possible that some settings below don’t “exist” yet. If the server refuses to (re)start and complains about an unknown setting, simply remove it/comment it out. Better yet, start planning a software update…)

First, back up your database. Really. Nothing here should be dangerous, but better safe than sorry. And, if possible, try out these changes on a test machine before moving to the real, production one.

For extra fun, run MySQLTuner now and save the results (e.g. mysqltuner.pl > ~/mysql-pre-optimization.txt), so you can compare them later…

So, edit (after backing it up, of course) your /etc/my.cnf (or /etc/mysql/my.cnf in Ubuntu, or /etc/my.cnf.d/server.cnf in Fedora), and look for the [mysqld] section.

You can add the following entries to the end of that section, they’ll replace their settings if they’ve already been previously set.

If you’re using InnoDB (and, between it and MyISAM, you certainly should):

innodb_buffer_pool_size = MEM1
innodb_log_file_size = MEM2
innodb_buffer_pool_instances = NUMBER

MEM1 should be about 80% of your available RAM (see “About memory” above). You can specify it as a number followed by “M” for megabytes, “G” for gigabytes, etc.;
MEM2 should be one eighth (1/8) of MEM1. For example, for 1 GB of MEM1, set MEM2 to 128M;
NUMBER should be the number of gigabytes (rounded down) of MEM1. For instance, for 4 GB, set NUMBER to 4. Note that this value doesn’t have a unit (M, G, etc.)

If you’re not using InnoDB at all (why?), then don’t add any of the above, of course.

For MyISAM:

key_buffer_size = MEM3

If your databases are all InnoDB, then leave this value small (e.g. 4M), as the MySQL user tables and such are still MyISAM. If you have many and/or large MyISAM tables, on the other hand, set MEM3 to 25% of the available RAM (see “About memory” above), and reduce the InnoDB memory by that amount (see MEM1 above). If you have only MyISAM tables, then you can raise MEM3 to 50%-80% of available memory.

If you’re using Aria tables, the setting is aria_pagecache_buffer_size; follow the key_buffer_size recommendations.

Other settings:

I find the following settings to work pretty well in most cases (and MySQLTuner seems to agree):

query_cache_type = 0 # recommended to be off, these days
query_cache_size = 0
thread_cache_size = 128
table_open_cache = 2048
low_priority_updates = 1 # MyISAM only, but no harm
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
tmp_table_size = 64M
max_heap_table_size = 64M

And that’s it! Restart MySQL, and you should enjoy much better performance, both for MySQL itself and for the rest of the server.

September 7, 2017April 9, 2018

How to configure TLS encryption in Dovecot

Dovecot As with most other internet services, Dovecot can be configured to use TLS encryption — and, unlike some others (such as web servers or SMTP servers), there’s little reason not to enforce it.

Notes:

Like other recent TLS tutorials on this blog, this is, of course, not a full Dovecot guide — that would be far too complex for a blog post. Instead, it’s just focused on enabling/configuring TLS encryption. I’ll be assuming you already have Dovecot configured, and are able to access mailboxes on your server using an email client.
I’ll be focusing on IMAP only, not POP3 or anything else.
The server will be configured to require encryption, for both privacy and security reasons. I don’t know of any modern email client that doesn’t allow encrypted connections (and even then, you might work around it by configuring an encrypted tunnel, but that falls out of this post’s scope).

1. Getting a certificate

While I believe most modern email clients will support ECDSA certificates, such clients are not a known quantity like, say, web browsers are, so I suggest that you create an RSA certificate, unless you already have one whose Common Name (CN) matches your server’s public name (e.g. mail.domain.com). Dovecot versions 2.2.31 and newer support configuring alternative certificates so that you could support both kinds at the same time, but there’s probably little gain in doing that, since most clients will likely default to RSA anyway (and, unless you’re using Let’s Encrypt, certificates are not free).

So, put your certificate and private key in /etc/dovecot/. Let’s assume the certificate is called myserver-full.crt ¹, and the private key is myserver.key. You should protect the private key from any users on your server, so do, for instance,

chown root:dovecot /etc/dovecot/myserver.key
chmod 640 /etc/dovecot/myserver.key

2. Configuring Dovecot for TLS

I’m assuming your Dovecot installation has a basic configuration file in /etc/dovecot/dovecot.conf, but the real “meat” of the configuration is in included files in /etc/dovecot/conf.d/. Your system may be slightly different, but I’m sure you can adapt. 🙂 Also, the numbers at the beginning of the file names may differ in your system.

This one isn’t really related to TLS, but it’s a good idea: edit /etc/dovecot/conf.d/20-imap.conf, and look for a line like this:

mail_plugins = $mail_plugins

and add to it (separated by a space):

imap_zlib

After all, there’s no reason not to use compression here, and bandwidth (especially on mobile) is still precious.

Now, edit /etc/dovecot/conf.d/10-ssl.conf, and add ²:

ssl = required
ssl_cert = </etc/dovecot/myserver-full.crt
ssl_key = </etc/dovecot/myserver.key
ssl_cipher_list = ECDHE-RSA-CHACHA20-POLY1305:ALL:!LOW:!SSLv2:!EXP:!aNULL
ssl_protocols = !SSLv2 !SSLv3
ssl_prefer_server_ciphers = yes

(The ssl_cipher_list line, besides setting secure defaults, sets the ChaCha20 protocol as the first one to be tried, since it’s considered one of the fastest and most secure. Note that it’ll require Dovecot linked to LibreSSL or OpenSSL 1.1.x to use that cipher, though Dovecot won’t complain if it doesn’t have access to it; it’ll just use the normal, secure defaults.)

If you don’t change anything else from the default configuration, the server will be listening on port 143 (IMAP) and port 993 (IMAPS). “But,” you ask, “isn’t standard IMAP unencrypted? I thought you were enforcing encryption…” Yes, but the standard IMAP port can be “upgraded” to TLS by entering the STARTTLS command, and email clients not only support that, but typically default to it (if not, just make sure you enable it). The server will refuse to authenticate any users on port 143 before they’ve “STARTTLSed”.

If you wanted to use only IMAP+STARTTLS, or only IMAPS, just edit /etc/dovecot/conf.d/10-master.conf, look for the “service” configuration, and disable the one you don’t want. But I see no problem with keeping both enabled.

Thoughts? Questions?

(To do the same to SMTP / Postfix, please see How to configure TLS encryption in Postfix.)

September 6, 2017April 9, 2018

How to create an RSA certificate

Whether it’s a web server, an email server (two, in fact, assuming you’re using one for SMTP and another for IMAP, as is common), or other types of applications, to have encrypted connections a TLS certificate ¹ is required. It can be self-signed (say, if you’ll be the only one needing to access that server), but browsers and email clients will complain (loudly!), therefore, if you want your server to be universally accessed, a “real” certificate is needed, so let’s show how to get one.

This recipe creates an RSA certificate (strongly suggested for, say, Postfix, since you can’t control what other email servers support, so you should go for the most common option); for web servers (accessed by standard browsers), an ECDSA certificate might be a good alternative.

A few notes:

There are alternatives (e.g. key sizes, etc.) for basically every parameter I’m using, but I’m not going into those. This is supposed to be as quick and easy as possible, after all.
Similarly, I’m not going into Let’s Encrypt; instead, I’m assuming a “normal” certificate authority such as Comodo. If you do use Let’s Encrypt, just use it to create the certificate instead of the “send the CSR to the certification authority” part.
The certificate (and server) will be compatible with most browsers and applications (email clients, etc.), but that “most” won’t include any Microsoft browsers on Windows XP (Firefox or Chrome on that abomination of an OS will still work).

Generating the key and the Certificate Signing Request (CSR):

openssl req -nodes -newkey rsa:4096 -keyout myserver.key -out myserver.csr

The command will ask you for details about your server/company (location, etc.). You should fill in every field, although the only mandatory one is “Common Name” (CN), which must match your server’s public name (not necessarily the machine’s name, but the hostname people will type in the browser/email client, such as “zurgl.com” or “mail.something.com“. Note that a certificate for “domain.com” also includes “www.domain.com” (so don’t include the “www.” in the CN), but if the server is reached at “subdomain.domain.com“, then that’s what the CN needs to be.

Ordering and receiving the new certificate:

Now go to a certification authority (CA), order a new certificate, and when asked for a CSR, send them (usually you can just copy and paste it into a text entry window) that myserver.csr file.

That file is, to all intents and purposes, your new certificate, and it’s ready to be used in application servers. You can use the same certificate for several services (e.g. an HTTPS website, an SMTP server, an IMAP server), as long as the hostname matches the certificate’s CN (so a certificate for “mail.myserver.com” won’t work for “www.myserver.com“, and vice-versa, but that’s because the names don’t match — not because they’re different services).

Adding TLS encryption (with your shiny new certificate) to internet services:

Here’s a (growing) list of tutorials:

(Note: most of this post’s content originally appeared in a previous one. The difference is that that one refers to ECDSA certificates, while this one is about the older, more common RSA certificates.)

September 6, 2017June 11, 2019

How to configure TLS encryption in Postfix

Although Postfix (and the SMTP protocol in general) can function without any kind of encryption, enabling TLS it can be a good idea in terms of both security and privacy, so let’s look at how it can be easily done.

We’ll actually be configuring two separate types of encryption:

Opportunistic encryption for regular SMTP (port 25), both incoming ¹ and outgoing ². “Opportunistic”, here, means that the server will ask for encryption and use it if the other side also supports it, but if it doesn’t then it’ll work without encryption. Although forcing it might sound tempting for security reasons, in reality many “smaller” servers around the world still don’t support it, so you would be unable to send or receive mail to/from those. Opportunistic TLS at least means that most “big” email services (e.g. Gmail) will communicate with you (and you with them) with encryption.
Mandatory encryption for the submission service (port 587). This is used by your own users (even if it’s just you) to send mail through your server, and typically has different restrictions (e.g. it’s authenticated, but on the other hand it can be used to send mail to the outside (unlike incoming port 25 which doesn’t, as that would make it an open relay)). Both the users’ authentication that would otherwise be in clear text, and the fact that only a limited number of users (your users, too) will be accessing that port through email clients (never other email servers that you don’t control) make forcing encryption here a no-brainer.

NOTE: naturally, this is not an exhaustive Postfix tutorial. I’ll be assuming you already have a working server, and just want to add TLS encryption to it. Also, some parts may not apply to your case.

1. Have/get a TLS certificate for your email host’s public name

If you don’t already have one, see how to create a new TLS certificate. We’ll be using an RSA certificate here (instead of going with ECDSA) since unlike, say, a web server accessed by standard browsers, you can’t control what other email servers support ³, therefore it’s best to choose the most common option. ⁴

Note that the certificate’s CN must match your email host’s public name (e.g. “mail.domain.com“). That will also probably correspond to (one of) the domain’s MX records.

2. Postfix configuration

Again, I’ll be assuming your non-TLS Postfix is already working fine.

Put your certificate and key in /etc/postfix (for instance). For this example, I’ll be calling them myserver-full.crt and myserver.key.

In /etc/postfix/main.cf, add the following lines:

# TLS configuration starts here

tls_random_source = dev:/dev/urandom

# openssl_path=/usr/local/libressl/bin/openssl
# uncomment and edit the above if you're using a different "openssl" than the system's
# (in this case, LibreSSL)

# SMTP from your server to others
smtp_tls_key_file = /etc/postfix/myserver.key
smtp_tls_cert_file = /etc/postfix/myserver-full.crt
smtp_tls_CAfile = /etc/postfix/myserver-full.crt
smtp_tls_security_level = may
smtp_tls_note_starttls_offer = yes
smtp_tls_mandatory_protocols=!SSLv2,!SSLv3
smtp_tls_protocols=!SSLv2,!SSLv3
smtp_tls_loglevel = 1
smtp_tls_session_cache_database =
    btree:/var/lib/postfix/smtp_tls_session_cache

# SMTP from other servers to yours
smtpd_tls_key_file = /etc/postfix/myserver.key
smtpd_tls_cert_file = /etc/postfix/myserver-full.crt
smtpd_tls_CAfile = /etc/postfix/myserver-full.crt
smtpd_tls_security_level = may
smtpd_tls_auth_only = yes
smtpd_tls_mandatory_protocols=!SSLv2,!SSLv3
smtpd_tls_protocols=!SSLv2,!SSLv3
smtpd_tls_loglevel = 1
smtpd_tls_session_cache_database =
    btree:/var/lib/postfix/smtpd_tls_session_cache

# TLS configuration ends here

And in /etc/postfix/master.cf, assuming you already have a “submission” section a bit like this:

submission  inet  n     -       n       -       -       smtpd
    -o smtpd_etrn_restrictions=reject
    -o smtpd_sasl_auth_enable=yes
    -o smtpd_recipient_restrictions=permit_mynetworks,permit_sasl_authenticated,reject
    -o smtpd_client_restrictions=permit_mynetworks,permit_sasl_authenticated,reject
    -o smtpd_helo_restrictions=permit_mynetworks,permit

Add to that section:

  -o smtpd_tls_security_level=encrypt

The above forces encryption for the submission service (remember that it’s not required for normal SMTP, it’s just desired).

In short:

your server tries to connect to others using TLS, but falls back to an unencrypted connection if the other side doesn’t support encryption;
other servers can connect to yours using TLS, but if they don’t attempt it then the connection will be unencrypted;
your users *must* use encryption (and authentication) to send mail through your server.

Restart your server, and check the logs: you should be getting mentions of TLS now. For instance, an email server starting an encrypted connection to yours looks like:

Sep 6 14:25:58 drax postfix/smtpd[22727]: Anonymous TLS connection established from lists.openbsd.org[192.43.244.163]: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)

Questions? Suggestions?

(And what about securing mailbox access? Enjoy: How to configure TLS encryption in Dovecot)

September 5, 2017April 9, 2018

How to create an ECDSA certificate

In this tutorial, I’ll be creating an ECDSA certificate with an EC key, instead of the more usual RSA type; ECDSA is more modern and is theoretically more secure even with smaller keys. If you need an RSA certificate (which I’d recommend for Postfix, for instance, since you can’t control what other email servers around the world support, so, you should go for the most common option; for web servers, ECDSA is fine), see How to create an RSA certificate.

A few notes:

There are alternatives (e.g. key sizes, etc.) for basically every parameter I’m using, but I’m not going into those. This is supposed to be as quick and easy as possible, after all.
Similarly, I’m not going into Let’s Encrypt; instead, I’m assuming a “normal” certificate authority such as Comodo (which supports ECDSA certificates; if you choose another, you should make sure of that in advance). If you do use Let’s Encrypt, just use it to create the certificate instead of the “send the CSR to the certification authority” part.
The certificate (and server) will be compatible with most browsers and applications (email clients, etc.), but that “most” won’t include any Microsoft browsers on Windows XP (Firefox or Chrome on that abomination of an OS will still work).

Generating the key and the Certificate Signing Request (CSR):

openssl ecparam -genkey -name secp384r1 | openssl ec -out myserver.key

openssl req -new -key myserver.key -out myserver.csr

The second command will ask you for details about your server/company (location, etc.). You should fill in every field, although the only mandatory one is “Common Name” (CN), which must match your server’s public name (not necessarily the machine’s name, but the hostname people will type in the browser/email client, such as “zurgl.com” or “mail.something.com“. Note that a certificate for “domain.com” also includes “www.domain.com” (so don’t include the “www.” in the CN), but if the server is reached at “subdomain.domain.com“, then that’s what the CN needs to be.

Ordering and receiving the new certificate:

Now go to a certification authority (CA), order a new certificate, and when asked for a CSR, send them (usually you can just copy and paste it to a text entry window) that myserver.csr file.

Adding TLS encryption (with your shiny new certificate) to internet services:

Here’s a (growing) list of tutorials:

(Note: most of this post’s content originally appeared in a previous one. The reason for its creation is that I’m planning several other TLS-related posts where a certificate will be required and, to avoid repeating the “how to create a certificate” instructions in every single one of them, I’d rather have this post to link to when needed.)

September 4, 2017April 9, 2018

How to set up an Nginx HTTPS website with an ECDSA certificate (and get an A+ rating on SSL Labs)

Most people who maintain web servers (or email servers, or…) have probably had to deal with SSL/TLS certificates in the past, but the process is, in my opinion, typically more complex than it should be, and not that well documented, typically forcing the user to consult several tutorials on the web so that they can adapt parts of each for their needs. Since I had to renew a couple of certificates last week, I thought about writing a quick and easy tutorial for a particular case: HTTPS on Nginx (although the certificate itself could, of course, be used for other services; right now I have one I’m using for IMAP (Dovecot) and SMTP (Postfix) as well). For extra fun, I’ll be creating/using an ECDSA certificate with an EC key, instead of the more usual RSA type; ECDSA is more modern and is theoretically more secure even with smaller keys.

A few notes:

There are alternatives (e.g. key sizes, etc.) for basically every parameter I’m using, but I’m not going into those. This is supposed to be as quick and easy as possible, after all.
Similarly, I’m not going into Let’s Encrypt; instead, I’m assuming a “normal” certificate authority such as Comodo (which supports ECDSA certificates; if you choose another, you should make sure of that in advance). If you do use Let’s Encrypt, just use it to create the certificate instead of the “send the CSR to the certification authority” part.
The certificate (and server) will be compatible with most browsers, but that “most” won’t include any Microsoft browsers on Windows XP (Firefox or Chrome on that abomination of an OS will still work).
If you already have a certificate (whether ECDSA or not, whether from Let’s Encrypt or not) and you’re here just for the A+ rating on SSL Labs, skip to “Setting up Nginx” below.

Generating the key and the Certificate Signing Request (CSR):

openssl ecparam -genkey -name secp384r1 | openssl ec -out myserver.key

openssl req -new -key myserver.key -out myserver.csr

The second command will ask you for details about your server/company (location, etc.). You should fill in every field, although the only mandatory one is “Common Name” (CN), which must match your server’s public name (not necessarily the machine’s name, but the host name people will type in the browser, such as “zurgl.com“. Note that a certificate for “domain.com” also includes “www.domain.com” (so don’t include the “www.” in the CN), but if the server is reached at “subdomain.domain.com“, then that’s what the CN needs to be.

Ordering and receiving the new certificate:

Now go to a certification authority (CA), order a new certificate, and when asked for a CSR, send them (usually you can just copy and paste it to a text entry window) that myserver.csr file.

If everything went well, then the CA should email you the new certificate in a short while. Typically they send you two files: the certificate itself, and a couple of “intermediate” certificates. Only the first is really needed, but I’ve had best results with concatenating your certificate (first) and the intermediate certs (last) into a single file, which you might call myserver-full.crt . Put it somewhere Nginx can access (e.g. /etc/nginx), and also the key you generated earlier (in this example, myserver.key). You don’t need the CSR there, by the way; it was just needed for ordering the new certificate.

Setting up Nginx:

This is, of course, not a complete Nginx tutorial (that would take a lot more space than a single post), just a simple recipe for configuring an HTTPS site with your new certificate, and have it be secure (and get a great score at SSL Labs, too). I’m assuming you can take care of all the non-HTTPS bits.

So, inside a virtual host, you need the server section:

server {
        listen 443 ssl http2; # if your nginx complains about 'http2', remove it, or (better yet) upgrade to a recent version
        server_name mysite.com # replace with your site, obviously

        # all the non-HTTPS bits (directories, log paths, etc.) go here

        # the rest of the recipe -- see below -- can go here
}

See the last comment? OK, let’s begin adding stuff there (I won’t indent any configuration lines from now on, but they look better if aligned with the rest of the configuration inside the { } ).

ssl_certificate /etc/nginx/myserver-full.crt; # the certificate and the intermediate certs, as seen above...
ssl_certificate_key /etc/nginx/myserver.key; # ... and the key

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

ssl_ciphers 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS:!3DES';
# the above comes from Mozilla's Server Side TLS guide

add_header X-Content-Type-Options nosniff;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 5m;	# basic defaults.

Great, you now have a basic HTTPS server! I suggest you try it now: save the configuration, restart Nginx, and test it. Access the URL with a browser (and check if the browser doesn’t complain about the certificate: if it did, something went wrong), and run it through SSL Labs’ Server Test. Check if it complains about something; if so, something needs fixing (ask in the comments, maybe I or someone else can help).

If all went well, you probably got a decent score, maybe even an A. But how to get an A+, like I promised at the beginning?

An A+ score on SSL Labs’ Server Test:

As of now (September 2017), SSL Labs only asks for one more thing: “HTTP Strict Transport Security (HSTS) with long duration”. HSTS is a mechanism to tell browsers: “this site should be accessed through HTTPS only; if you attempt to connect through normal HTTP, then deny access”. This information is actually cached by the browser, it’s not the server that refuses HTTP connections (though it’s, of course, possible to configure Nginx to do just that — or simply do 301 redirects to the equivalent HTTPS URL. But I digress…). In short, it protects against downgrade attacks. As for the “with long duration” part, that’s simply the time that browsers should cache that information.

IMPORTANT: don’t proceed until you have the basic HTTPS site running, and SSL Labs reporting no problems (see the previous section)!

ALSO IMPORTANT: if you *do* want your site to be accessible through both HTTP and HTTPS, stop here, as the rest of the configuration will make it HTTPS-only! That means, however, giving up on the A+ score.

So, add the following:

add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";

Restart Nginx, try SSL Labs’ test again. Did you get an A+?

If not, or you have any questions, feel free to ask.

For extra fun: have your Nginx use LibreSSL or OpenSSL 1.1.x, and enjoy a few more modern ciphers (look for “ChaCha20” on SSL Labs) without any configuration changes from the above.

September 2, 2017April 9, 2018

Linux: How to Compile Stuff

A couple of days ago I was surprised when someone who’s worked as a Linux sysadmin for several years asked me how to compile a program that was only available in source format (it was a cryptocurrency miner, for the curious), since he’d never compiled anything in his life. How was that possible? But then I realized: compiling stuff on Linux hasn’t really been a necessity ¹ for the last, oh, 15-20 years or so. Back in 1993 or so (yikes!), when I started using Linux, things were of course quite different.

So, without further ado: how to compile stuff on Linux. Assuming you’ve already downloaded and uncompressed a program’s source, here’s what works 99% of the time (from the program’s directory, of course):

./configure --help  # to see available options

./configure # followed by the desired options, if any

make && make install

By default, compiled programs are then installed to /usr/local (e.g. binaries go to /usr/local/bin, libraries to /usr/local/lib , etc.). If you want to change that, just add –prefix=path (e.g. –prefix=/usr/local/test) to the ./configure options. (In that example, it’ll install binaries to /usr/local/test/bin, and so on. There are ways (i.e. different ./configure options) to install to different sub-paths, but those are beyond this basic tutorial. The default /usr/local/bin, lib, etc. paths also have the advantage of being in the default executable and library paths in typical distributions, so that you don’t have to add to those.)

If the above commands worked for you, congratulations, you’ve compiled your first program, which should now be ready to run! 🙂 However, if you’ve never compiled anything before on that system, it’s more likely that the ./configure command complained about missing libraries, missing utilities, or even a missing compiler; after all, as the first paragraph showed, it’s possible to use Linux for years, even work with it for a living, without needing to compile anything, which in turn has caused most modern distributions to not install compilers and development stuff by default.

How to do so? On Debian/Ubuntu, start with:

apt-get install build-essential

And on Red Hat/CentOS/Fedora, use:

yum groupinstall “Development Tools”

The above should install the C/C++ compiler and common development tools. However, a particular program may ask for other libraries’ development files, which may not be installed even though the library itself is. It’s impossible to give you an exhaustive list, but typically you can use yum search or apt-cache search to find what you need. Debian-based development packages typically end in -dev, while Red Hat-based ones end in -devel . For instance, suppose you’re on Ubuntu and ./configure complains about not having the zlib development files. A quick search would point you (at least on Ubuntu 17.04) to zlib1g-dev, which you’d install with apt-get install zlib1g-dev .

And what if there is no configure script (unlikely these days, but possible if it’s a very old source)? If there is a file named Makefile, then try typing make, and see if it compiles. Otherwise, your best bet is to read any provided documentation (e.g. a README or INSTALL file).

Any questions, feel free to ask.

August 31, 2017April 9, 2018

Ubuntu/Debian: Installing Nginx/Postfix/Dovecot using OpenSSL 1.1.x

OpenSSL So, let’s say you have a Ubuntu or Debian server, using one or more of Nginx, Postfix, and Dovecot, and you’d like to have them link to OpenSSL 1.1.x instead of the default OpenSSL (as of Ubuntu 17.04, it’s version 1.0.2g). (Reasons may include wanting to use modern ciphers such as ChaCha20, or trying out support for the most recent TLS 1.3 draft. Also, if you want to try out LibreSSL instead of OpenSSL 1.1, please check out the previous post.)

So, here’s a relatively simple way, that doesn’t change the system’s default OpenSSL (believe me, that wouldn’t be a good idea, unless you recompiled everything):

Install dependencies:

apt-get install build-essential
apt-get build-dep openssl nginx dovecot postfix

Install OpenSSL 1.1.x:

download the latest 1.1.x source from www.openssl.org
compile and install it with:

./config --prefix=/usr/local/openssl11 --openssldir=/usr/local/openssl11 && make && make install

Install Nginx:

rm -rf /usr/local/src/nginx-openssl11
mkdir /usr/local/src/nginx-openssl11
cd /usr/local/src/nginx-openssl11
apt-get source nginx # ignore the permissions error at the end

edit nginx-<version>/debian/rules: add

–with-openssl=/usr/local/src/openssl-

to the beginning of the common_configure_flags option (note that that’s the source directory you used to compile OpenSSL 1.1.x, not where you installed it to);

debuild -uc -us -b

install the required packages in the parent directory with “dpkg -i ” (do dpkg -l | grep nginx to see which you have installed; typically you’ll want to install the newly created versions of those);

apt-mark hold nginx* # to prevent nginx from being updated from Ubuntu / Debian updates

Done! You can now play around with the Mozilla TLS Guide to add support for modern ciphers to your Nginx’s configuration, and use SSLLabs’s SSL Server Test tool to check if they are correctly enabled.

Install Postfix:

It’s just like Nginx (replacing “nginx” with “postfix” in every command / directory name, of course), except that the changes to debian/rules are these:

find -DHAS_SSL, add -I/usr/include/openssl11/include/openssl in front of it;
find AUXLIBS += , add -L/usr/local/openssl11/lib in front of it
find the line with dh_shlibdeps -a, add –dpkg-shlibdeps-params=–ignore-missing-info to it
don’t forget the apt-mark hold postfix* at the end.

Install Dovecot:

Again, use the Nginx instructions, using “dovecot” instead of “nginx” everywhere, except that the changes to debian/rules should be:

after the line:

export DEB_BUILD_MAINT_OPTIONS=hardening=+all

add:

export SSL_CFLAGS=-I/usr/local/openssl11/include
export SSL_LIBS=-L/usr/local/openssl11/lib -lssl -lcrypto

after the section:

override_dh_makeshlibs:
# Do not add an ldconfig trigger; none of the dovecot shared libraries
# are public.
        dh_makeshlibs -n

add:

override_dh_shlibdeps:
        dh_shlibdeps --dpkg-shlibdeps-params=--ignore-missing-info

NOTE: the indentation in the second line needs to be a tab, don’t use spaces.

Again, remember to apt-mark hold dovecot* after installation.

How to check if your new installations of Postfix and/or Dovecot are using OpenSSL 1.1.x instead of the default OpenSSL 1.0.x? You could use ldd to check what SSL / TLS libraries your binaries and/or libraries link to, but the best way is probably to use a tool such as sslscan, which you can use to check what ciphers your SMTP, IMAP, etc. support (including with STARTTLS). If you see ChaCha20 in there, everything is fine. 🙂

If you ever want to go back to “normal” versions of these servers, just do apt-mark unhold nginx* (for instance).