Bash Archives

Let’s say you want to move a (relatively large) filesystem, with an application’s data files (including a lot of subdirectories, many of them with specific ACLs), to a faster disk (for instance). The old FS is mounted as /PROD, while the new disk is mounted as /NEW; the idea is to stop the application, rsync the data, and switch mount points. Easy, you think:

[pastacode lang=”bash” manual=”rsync%20-av%20%2FPROD%2F%20%2FNEW%2F” message=”” highlight=”” provider=”manual”/]

So, the application guys stop the app, you enter the command above, then switch mount points so the old /NEW is now /PROD, and the old /PROD is now /OLD . Everything looks fine, the application guys start their thing again…

… and then complain about access problems. Oops, the rsync didn’t copy the ACLs — you needed the -A argument to include them (e.g. rsync -aAv).

But they took a while to notice it, so that there’s a lot of new data in the new filesystem, so simply doing a new rsync from /OLD to /PROD is unacceptable. You’d simply like to copy directory ACLs (for simplification’s sake, let’s assume there are no file ACLs here — but see below). But there are a couple thousand directories, and of course you don’t want to do it manually…

The solution I came up with was:

[pastacode lang=”bash” manual=”cd%20%2FOLD%0Afind%20.%20-type%20d%20%7C%20cut%20-d%20′.’%20-f%202-%20%3E%20%2Ftmp%2Fdirectories.txt%0AIFS%3D%24’%5Cn’%20%3B%20for%20i%20in%20%60cat%20%2Ftmp%2Fdirectories.txt%60%3B%20do%20getfacl%20%22%2FOLD%24i%22%20%7C%20setfacl%20–set-file%3D-%20%22%2FPROD%24i%22%20%3B%20done” message=”” highlight=”” provider=”manual”/]

Basically, it stores a list of directories in /OLD , then, for each of them, it copies the ACL from /OLD/<directory> to /PROD/<directory> . setfacl’s “–set-file=-” means use the standard input as a “file”, which comes from getfacl‘s output. The “IFS=$’\n’” bit means that the for loop cycles through entire lines, not “words” — otherwise, it would try to split paths with spaces in them.

A limitation of this is that any newly created directories in the new /PROD filesystem won’t be affected, but hopefully they won’t be too many.

What if you wanted to include files as well? Just remove the “-type d” in the find command. Note that, in this case, it’s much more likely that there are new files in the new /PROD FS that won’t have their ACLs corrected.

Let’s say you realize (maybe because you got an alarm for it) that a particular filesystem — let’s say /qwerty — is full, or almost full, and you want to find out what’s taking up the most space. Simply enter something like:

[pastacode lang=”bash” manual=”du%20-k%20-x%20%2Fqwerty%20%7C%20sort%20-n%20%7C%20tail%20-20″ message=”” highlight=”” provider=”manual”/]

to get a list of the directories taking up the most space in that filesystem, sorted by size, with the largest ones at the bottom.

Note that the “/qwerty“, in this case, should be the filesystem’s top directory, not some subdirectory of it. In other words, it should be something that shows up on a “df” command.

Explanation:

“du -k -x” shows the subdirectories (and their total sizes) of the specified directory (or of the current one, if you don’t specify one). “-k” means that sizes are reported in kilobytes (KB) — this is not mandatory, but different versions of “du” may use other units, and this one is easy to read. “-x” means “don’t go out of the specified filesystem”, and that’s quite important here, as you could have a “/qwerty/asdfgh” filesystem, mounted in a subdirectory of “/qwerty“, but since it’s a different filesystem (appearing separately in the results of a “df” command) you won’t want to include it in this list.

“sort -n” is a simple numeric sort, and “tail -20” just means “show only the last 20 lines. 20 is a reasonable number that fits in most terminals at their default sizes (typically 80×25), so that you don’t have to scroll up.

(Yes, this is very basic, but, hey, one of the goals of this blog is to document stuff I help co-workers with, since, at least in theory, if someone has a question about something they need to do, then many other people may have the same doubt/need as well, and we have a couple of new team members whose backgrounds are not Linux/Unix-related, so…)

Tag: Bash

Linux: “Oops, forgot to copy directory ACLs in an rsync…”

Linux: What’s filling up my almost full filesystem?

Explanation: