Waff.co

Linux ate my memory.

A mystery is afoot!

Once in a while I actually come across interesting problems. A server reporting a very high memory usage statistic, yet, no processes to be found consuming it.

Yeah, I'm not dumb over here, something's lying, I just need to figure out where it's all gone.

First to give some background, I'll throw out some generic information here about the system, but I'm going to butcher it, cause I didn't keep it from the actual machine. So don't expect the numbers to be perfect.

 free -m
              total       used       free     shared    buffers     cached
Mem:          2048       1638        410          0        93       124
-/+ buffers/cache:       1590        234
Swap:         512        0           512

Generally, high real used memory, and low cached, low free, no swap usage. From this you might normally expect that if you checked PS, that you would see a bunch of large processes eating memory ( because memory is yummy ). But what if you were wrong? (oh, so very wrong.) Checking PS, not a single process is using more than 0.5% memory, and even then, there's only one or two of them. This would be about expected, because the server in question only handles cron jobs (which only run about once every 10 minutes).

Wat?

So. You've seen, Linux ate my memory. It's just gone, all holed-up somewhere in there. Well, you also likely have a little program called slabtop that you can use. Slabtop is like top... but for kernel slab cache (see, it's quite aptly named). Now, I'm no kernel dev, but I'll take a swing at explaining wtf a "slab" is, and why you care (I'm not your mom, I can't make you care, pls no sue if I'm wrong).

A Slab (in the kernel context) is used for faster creation/destruction of kernel objects. It is one larger contiguous space that is pre-allocated for use with a homogenous group of objects. This means that when the kernel needs to make a new object, the slab allocator can just hand off a pointer to some pre-allocated memory. Since all the objects are the same type (and thus, size), a new object can slot right in over a previously-deleted object. Pretty neat, so any object creation or deletion is fairly quick when it's using this method.

That's only really tangentally related, in so much as the objects we seek are likely kernel objects. I'm going to avoid the deeper details of them, cause it's not necessary.

The caches we're talking about here are Filesystem-related. There are a few types, namely superblock, inode, dentry, and (most commonly) file. The definitions of each, I'll blurb about for the kids in the back.

  • superblock, is the metadata for the actual on-disk filesystem. It's recorded a few times on your physical disk, for redundancy; because if this was corrupted or lost, well your disk is just a big ol' bowl of binary soup now. Pretty friggin useless. It's kept in-memory because it's used a lot by the system while it's running, like space available, usage bitmaps, etc. If you want to see the spew, run dumpe2fs /dev/<block_device on a formatted drive on a system.

  • inode, which is metadata (creation time, size, location on disk, permissions, etc) about a particular file. If you're looking at the inodes of files a lot, they'll prolly end up in a cache, so the system doesn't have to hit disk to read this time amount of data so often.

  • file, pretty much exactly what it says on the tin, it's a cache of a file's contents.

  • dentry, is directory-entry cache. The others make sense, this one a bit not so much. These are sort of glue between inodes and file names. Each directory is basically a list of it's contents, all the names of the files, and their inodes. It's like a tree of entries with filenames and inodes, so (once again) the system doesn't have to hit the disk to get structure information about the filesystem.

In this case, slabtop (sorted by size) was really quick to show that dentry cache was the vast majority (actually, it was using like 1.4GB of memory). So, that was the usage, all of it.

If you've been using Linux for a bit, you likely know about it's file caching. This is marvelous and makes your system faster by not hitting disk. The kernel steals some system memory you're not currently using, and puts the contents of some files you're using a lot in there; RAM is faster than disks afterall. Generally if you have a lot of files cached in memory, you see a high cached amount in free, though this isn't the case.

How do I make the bad words go away?

Now, how to fix this? Well, first obviously the machine is doing something strange, if I had a guess about what it was doing; I would say that the crons running on it were doing a LOT of file listing or directory traversal. I don't know exactly, because I didn't write the script it is running. You could restart the machine, but that's a bad choice (you should feel bad). You could rewrite the script, which is maybe a non-option. The option i chose was setting the sysctl vm.vfs_cache_pressure to 1000 ( from 100 ), and run echo 2 > /proc/sys/vm/drop_caches to drop inode and dentry caches.

That should instantly clear the "extra" memory usage, and make Linux puke your memory back out.