Linux: What’s filling up my almost full filesystem?

(Yes, this is very basic, but, hey, one of the goals of this blog is to document stuff I help co-workers with, since, at least in theory, if someone has a question about something they need to do, then many other people may have the same doubt/need as well, and we have a couple of new team members whose backgrounds are not Linux/Unix-related, so…)

Let’s say you realize (maybe because you got an alarm for it) that a particular filesystem — let’s say /qwerty — is full, or almost full, and you want to find out what’s taking up the most space. Simply enter something like:

du -k -x /qwerty | sort -n | tail -20

to get a list of the directories taking up the most space in that filesystem, sorted by size, with the largest ones at the bottom.

Note that the “/qwerty“, in this case, should be the filesystem’s top directory, not some subdirectory of it. In other words, it should be something that shows up on a “df” command.


du -k -x” shows the subdirectories (and their total sizes) of the specified directory (or of the current one, if you don’t specify one). “-k” means that sizes are reported in kilobytes (KB) — this is not mandatory, but different versions of “du” may use other units, and this one is easy to read. “-x” means “don’t go out of the specified filesystem”, and that’s quite important here, as you could have a “/qwerty/asdfgh” filesystem, mounted in a subdirectory of “/qwerty“, but since it’s a different filesystem (appearing separately in the results of a “df” command) you won’t want to include it in this list.

sort -n” is a simple numeric sort, and “tail -20” just means “show only the last 20 lines. 20 is a reasonable number that fits in most terminals at their default sizes (typically 80×25), so that you don’t have to scroll up.

Networking definitions: Allowing traffic, Port forwarding, NAT, and Routing

I’m more of a systems administrator than a network admin (though I’ve also worked as the latter, in the past), but, of course, one can’t be a sysadmin without knowing at least something about networking. And yet (and like my previous post about compiling stuff), I’ve found that it’s possible to do a perfectly good job as a sysadmin, and yet still mix up a few networking concepts from time to time.

Therefore, I wanted to write about four different, but related, concepts, that I’ve noticed people sometimes confuse.

1. Allowing traffic

This just means that a network interface allows traffic to a specific destination and/or port, possibly restricted to a specific source (an IP address, or a network). Any traffic that is not allowed is simply refused.

Note that this by itself doesn’t mean that the interface (of a server, a router, etc.) will do anything (or anything desirable, at least) with the received traffic. For instance, it may not have something listening on that port, or it may not be configured to route that traffic to somewhere useful. This just means that it doesn’t instantly block the connection.

This is typically controlled by a local software firewall such as iptables.

2. Port forwarding

Port forwarding just means: if you receive a packet on interface A, port B, then redirect it to IP address C, port D. “D” may be the same as “B”, and “C” may be on the same host or at the other end of the planet.

Again, note that the mere fact that there’s a matching redirect rule for the initial destination interface and port doesn’t mean that the host actually accepts redirecting the traffic (see 1.) or that it knows how to route it to its final destination (see 4.)

3. NAT

NAT, or Network Address Translation, can be seen as a special case of two-way port forwarding (see 2.). Basically, a router (which may well be a simple server with two or more (physical or virtual) network interfaces, it doesn’t have to be a “router” bought in a store) accepts traffic from a (typically private) network, then translates it so that it goes to the destination address in the (typically public) network, with (and this is the important part) the source “masked” as the router’s public IP address, and a different source port that the router “remembers”, and then knows how to handle the returning traffic and “untranslate” it so that it goes to the original source.

In short, NAT allows many private hosts in a network to access the Internet using a single public IP, and a single connection. (It can have other similar uses, even some not related to the public Internet, of course; this is just the most common one.)

4. Routing

Routing is simply a host knowing that a packet intended for IP address X should be directed to IP address Y 1.

Again, the obligatory caveats: this doesn’t mean that the traffic is even accepted before attempting to route it (see 1.), or that the host actually knows how to reach IP address Y itself (it may not have a local route to it). It may also be that the routing is correct, but there is no address translation (see 3.), so the destination host receives a packet in a public interface that claims to be from a private address, and refuses it. Finally, the destination host itself needs a route to the original source, and it may not have one configured (or have a misconfigured one, causing asynchronous routing and possibly making the source refuse the returning traffic).

Thoughts? Corrections? Clarifications? Yes, I know this is relatively basic stuff. 🙂

Linux: How to Compile Stuff

A couple of days ago I was surprised when someone who’s worked as a Linux sysadmin for several years asked me how to compile a program that was only available in source format (it was a cryptocurrency miner, for the curious), since he’d never compiled anything in his life. How was that possible? But then I realized: compiling stuff on Linux hasn’t really been a necessity 2 for the last, oh, 15-20 years or so. Back in 1993 or so (yikes!), when I started using Linux, things were of course quite different.

So, without further ado: how to compile stuff on Linux. Assuming you’ve already downloaded and uncompressed a program’s source, here’s what works 99% of the time (from the program’s directory, of course):

./configure --help  # to see available options

./configure # followed by the desired options, if any

make && make install

By default, compiled programs are then installed to /usr/local (e.g. binaries go to /usr/local/bin, libraries to /usr/local/lib , etc.). If you want to change that, just add –prefix=path (e.g. –prefix=/usr/local/test) to the ./configure options. (In that example, it’ll install binaries to /usr/local/test/bin, and so on. There are ways (i.e. different ./configure options) to install to different sub-paths, but those are beyond this basic tutorial. The default /usr/local/bin, lib, etc. paths also have the advantage of being in the default executable and library paths in typical distributions, so that you don’t have to add to those.)

If the above commands worked for you, congratulations, you’ve compiled your first program, which should now be ready to run! 🙂 However, if you’ve never compiled anything before on that system, it’s more likely that the ./configure command complained about missing libraries, missing utilities, or even a missing compiler; after all, as the first paragraph showed, it’s possible to use Linux for years, even work with it for a living, without needing to compile anything, which in turn has caused most modern distributions to not install compilers and development stuff by default.

How to do so? On Debian/Ubuntu, start with:

apt-get install build-essential

And on Red Hat/CentOS/Fedora, use:

yum groupinstall “Development Tools”

The above should install the C/C++ compiler and common development tools. However, a particular program may ask for other libraries’ development files, which may not be installed even though the library itself is. It’s impossible to give you an exhaustive list, but typically you can use yum search or apt-cache search to find what you need. Debian-based development packages typically end in -dev, while Red Hat-based ones end in -devel . For instance, suppose you’re on Ubuntu and ./configure complains about not having the zlib development files. A quick search would point you (at least on Ubuntu 17.04) to zlib1g-dev, which you’d install with apt-get install zlib1g-dev .

And what if there is no configure script (unlikely these days, but possible if it’s a very old source)? If there is a file named Makefile, then try typing make, and see if it compiles. Otherwise, your best bet is to read any provided documentation (e.g. a README or INSTALL file).

Any questions, feel free to ask.

Linux: find users with total sudo access on many machines

Disclaimer: there are surely many, far better ways to do this — feel free to add them in the comments. This was just a quick and dirty script I came up with yesterday, after a co-worker wondered if there was an easy way to do this on all the servers we administer.

The situation: you administer 1000 or more servers, you and your team are the only users who are supposed to be able to sudo to root (unlike simply running certain specific commands, which is typically OK), but sometimes you have to grant temporary access to a particular user or group of users who, for instance, are doing the initial application installations, but who are supposed to lose that access when the server enters production.

The problem: it’s easy to forget about those, and so the temporary access becomes permanent (yes, there are other ways around that, such as using a specific syntax for those accesses that includes a comment that you then use a script, called by the “at” daemon, to remove later, but bear with me for now). Wouldn’t it be useful to be able to look at a group of some, or even all, of the servers you administer, and find those unwanted, forgotten sudo accesses?

Continue reading “Linux: find users with total sudo access on many machines”