Linux: ‘top’ shows almost all of my memory being used! Should I panic?

Top The short answer is, of course, “no.” 🙂 1

A common mistake that Linux users make — indeed, one that even some less experienced Linux sysadmins sometimes make — is treating Linux2 as if memory technology hadn’t evolved since MS-DOS, when you needed as much memory as possible — especially conventional RAM (anyone remember that?) — to be free, so that applications could be loaded into it and run. If there wasn’t enough free memory, then applications wouldn’t run, or would run worse, so you’d have to “optimize” your DOS system to have as much free conventional memory as possible (using tricks such as loading device drivers into upper memory, and, of course, exiting a program completely before starting another.)

Somehow, that line of thinking still exists — even, apparently, in people who never used MS-DOS or similar systems before(!). “Free memory is good,” people think. “If most of the RAM is used up, then there’s a problem,” they insist — one to be solved either by killing applications or by adding more memory to the system.

Linux — like all modern systems, and I’ll even include Windows here 🙂 3, uses virtual memory. Put simply (and, yes, I’m simplifying it a lot — there’s more to it, of course), it means that the amount of available memory is actually (mostly) independent of the physical system RAM, since the system uses the disk as memory, too. Since real RAM is (virtually) always faster — but (usually) smaller — than a disk drive, the system typically uses prefers to use it for more critical stuff (such as the kernel, device drivers, and currently actively running code), with less critical stuff (say, a program waiting for something to happen) being relegated to the disk — even though it’s still “memory”. Note that even most of the aforementioned “critical stuff” can be temporarily moved to the disk if needed.

You may now be asking: if RAM is typically preferred for “critical stuff” only, and that stuff doesn’t take up the entire available memory, then what’s the rest of the RAM used for? The answer is, of course, disk caching — both for reading (loading up recently and/or frequently accessed data from the disk and keeping it available in RAM for a while, in case it’s needed again in the future) and for writing (which works the other way around — the system tells the program the data is already written to disk, so that it can go on instead of waiting for it, but it’s still only in memory, to be written a short while later.) So, any free RAM is typically used for disk caching, with the system managing it automatically.

But people keep looking at the output of “top”, seeing 90% of the physical RAM used, and panicking. 🙂

In fact, this confusion was so common that modern Linux systems even changed some labels on the “top” command. For instance, this is from a Red Hat Enterprise Linux 6.x:

"top" on RHEL 6.x
“top” on RHEL 6.x. Nice machine, eh? 🙂

As seen above, the server has (rounding down) 567 GB of physical RAM, of which 506 GB — i.e. 89% — are used. Looks scary (if you haven’t been reading so far), right? I mean, the server is at almost 90% capacity! But then you look at the value for “cached”, in the line below. That’s 479 GB — or 84% — used for disk cache! In other words, the entire operating system and applications are running in just 5% of the server’s RAM, with the rest of it being used just to make disk access faster.

And that “cached” value — and this is the important bit — counts as free memory. It’s available to the system whenever needs it. In “computer terms”, it’s not technically free, it’s being used for the cache, but in “human terms”, it’s just as free as the one that actually says “free” in front of it.

As I said above, this caused so much confusion to users (for decades) that, eventually, it was changed. Here’s how it looks like in a RHEL 7.x system:

Top in RHEL 7.x
“top” in RHEL 7.x. Puny machine, I know.

In the first line: 79044 KB free + 240444 KB used + 697064 KB for buffers and cache (which were separated in the old version, by the way) = … 1016512 KB, which you’ll note is exactly the first value in the same line, for total physical memory.

In the second line, the first three values refer to swap, so they’re not relevant here, but the “avail Mem” one is new. What is it? According to the “top” manual page,

The avail number on line 2 is an estimation of physical memory available for starting new applications, without swapping. Unlike the free field, it attempts to account for readily reclaimable page cache and memory slabs.

In other words (and this is again an oversimplification), it’s what the system has readily available — in other words, the “free” part, plus (most of) what is being used for caching. That doesn’t mean it can only use that much memory, it just means that that’s what it can use right now, without swapping anything out. Which, in my opinion, ends up not mattering that much, in terms of a precise value. However, at least it now gives a better (though rough) idea of what part of the physical memory is available, not simply “free” (i.e. not doing anything).

So, what about those specific cases mentioned (in a note) at the beginning? Well, if there is very little memory free and very little swap space free and very little memory (compared to the total amount of physical memory) is being used for buffers/cache, then and only then4 may it be the case that the server actually needs more memory and/or swap space, or that some process has some kind of memory leak or is otherwise using much more memory than usual (if restarting the service fixes it, then this is probably the case.)

Linux: How to increase the size of an ext2/3/4 filesystem (using LVM)

(Insert obligatory disclaimer about this being basic stuff, but this blog being — among other things — about documenting stuff I’m asked about/help others with at work, because it may be useful to other users, etc. etc.)

If you work, or have worked, as a sysadmin, you’ve probably had to do this in the past, but, even on your own system — if you were wise enough to select “use LVM” during installation — you may find yourself facing a common problem: needing to grow a filesystem, such as / (the root filesystem), /opt, etc..

First, in many cases, you may not actually need to grow an existing filesystem: if you will need to store a lot of data in a specific directory (let’s say /opt/data, but /opt is currently almost full), then simply creating a new partition and mounting it (possibly mounting it first in a temporary location so that you can move the existing directory into it) as (for the above example) /opt/data 1, may well solve your problem. In this case, by the way, you don’t even need to be using LVM. You’ve just freed space on /opt and (depending on the new partition’s size) you now have a lot more room to grow on /opt/data as well.

But, since we want to challenge ourselves at least a little bit, let’s say we really need to grow that filesystem. As a requisite, the current disk must be a Logical Volume (LV) of an existing Volume Group (VG), made up of one or more Physical Volumes (PVs). Let’s also assume the filesystem you want to grow is /opt, that it’s an ext4 filesystem 2, and that it’s an LV named “lv_opt“, which is part of a VG called “vg_group1“, which currently has no free/unused space (if it did have enough unused space for your needs, you could skip to step 4 below.)

The process has several steps:

1- Add the new disk (or disks, but let’s say it’s just one for simplicity’s sake) to the system. This may mean adding a physical disk to a machine, or adding a new drive to a VM, or the company’s storage team presenting a new LUN, or… Anyway, the details of the many possibilities go beyond the scope of this tutorial, so let’s just assume that you have a new disk on your machine, and that Linux sees it (possibly after a reboot, in the case of a non-hotpluggable physical disk), as something like /dev/sdc (which we’ll use for the rest of the examples.)

2- Create the new PV: if you’re going to add the entire sdc disk to the VG, just enter:

pvcreate /dev/sdc

Otherwise, use fdisk to create a new partition inside it (sdc1), with the desired size, and change it to type “LVM” (it’s “8e” on fdisk). After exiting fdisk (saving the current configuration), enter (this is apparently not mandatory on modern Linux versions, but there’s no harm in it):

pvcreate /dev/sdc1

3- Add the new PV to the existing VG:

vgextend vg_group1 /dev/sdc # or /dev/sdc1, if the PV is just a partition instead of the entire disk

If you enter a “vgs” command now, you should see the added free space on the VG.

4- Extend the LV: if you want to add the entirety of the VG’s currently unused space, enter:

lvextend -l +100%FREE lv_opt

If, instead, you want to specify how much space to add (say, 10 GB):

lvextend -L +10G lv_opt

Note that one version of the command uses “-l” (lower case L) and the other uses “-L“. That’s because one refers to extents, while the other refers directly to size.

Now the LV has been extended, but wait! You still need to…

5- Extend the filesystem itself: just enter:

resize2fs /opt

Note that it’s a “2” in the command name, regardless of whether the filesystem is ext2, 3 or 4. This operation may take a while if the filesystem is large enough, but after it’s completed you should be able to see the new filesystem’s larger size (and added free space) with a “df” command.

Any questions or suggestions, feel free to add a comment.

Linux: Creating and using an encrypted data partition

Encrypted disks and/or filesystems are nothing new on Linux; most distributions, these days, allow you to encrypt some or all your disk partitions, so that they can only be accessed with a password. In this tutorial, however, we’re going to add a new encrypted partition to an existing system, using only the command line. I’ve found that tutorials on the web seem to make this issue more complex than it actually is, so here’s mine — hopefully it’ll be easier than most.

In this guide, we’ll be making a few assumptions. First, as said above, it’ll be an existing system, to which we’ve just added a new disk, /dev/sdb . Adapt to your situation, of course. If you’re not using an entire disk, just create a new partition with fdisk (e.g. /dev/sdb1) and use it instead. Second, we’ll have the machine boot with the disk unmounted, then use a command to mount it (asking for a passphrase, of course), and another to dismount it when you don’t need it. The obvious usage of such a partition is for storing sensitive/private data, but theoretically, you could run software on it — as long as you don’t automatically attempt to start it on boot, and don’t mind keeping it mounted most or all of the time, which perhaps defeats its purpose: if it’s mounted, it’s accessible.

So, without further ado…

Continue reading “Linux: Creating and using an encrypted data partition”

Today I Learned: Unix / Linux groups can have passwords

(Welcome to a new section on ZurglToday I Learned. As the name suggests, it’s for sharing Linux-related things I’ve just learned, even though I’ve been using Linux for over 20 years. Some of them may well be pretty basic (just like the following one) and even well-known; still, the “fun” part is that I’ve been able to work as a Linux sysadmin for two decades and administer several personal servers and use it as a desktop from time to time, and still hadn’t had a need for this until now.)

Did you know groups (not users) can have passwords, too? By default they don’t, but the groupadd command has a “-p” option (that requires an already encrypted password, so you’d need to encrypt it first and pipe it there). There’s also a gpasswd command. And, yes, an /etc/gshadow file.

Supposedly, the purpose of group passwords is for users to be able to join a password-protected group with the newgrp command, as long as they enter the group password correctly. If the group doesn’t have a password, then only someone with root access can add a user to it.

(newgrp also allows a user to change their own primary group for the duration of a session, as long as it’s one of their supplementary groups.)