More on file-nr, file-max

Not to be satisfied with not understanding my "Too many open files in system" problem, I started digging into the Linux 2.6.8 kernel source code.

kernel/sysctl.c defines the /proc/sys/fs files. There's fs_table, an array of ctl_table structs, that defines file-nr and file-max, and establishes that they're pointers to the files_stat structure, i.e. files_stat holds the data that's displayed when you 'cat file-nr'. file-nr is R/O, file-max is R/W.

fs/file_table.c (and include/linux/fs.h) defines files_stat. As you'd expect, it's a struct that holds nr_files, nr_free_files and max_files members. max_files is what gets changed when you alter /proc/sys/fs/file-max.

As described in Documentation/filesystems/proc.txt, nr_free_files is always zero as of the 2.6 kernel. max_files starts out at 10% of memory, but as mentioned, can be changed via /proc/sys/fs/file-max. Examples exist of altering startup scripts to do this at boot time, overriding the default where needed.

fs/file_table.c has get_empty_filp() that does the magic check to see if nr_files > max_files, and if so (and you're not root), deny you the new file. Failing this way puts an info message "VFS: file-max limit reached" into the logs.

fs/file_table.c is the only place that alters nr_files. It gets incremented in filp_ctor() and decremented in filp_dtor. filp_ctor/filp_dtor, in turn, get set as function pointers in fs/dcache.c when the "filp" kmem cache (named filp_cachep) gets created. filp_cachep, in turn, is also only referenced in fs/file_table.c, by get_empty_filp() and file_free().

get_empty_filp() and file_free() call kmem_cache_alloc() and kmem_cache_free(), respectively. These functions live in mm/slab.c.

file_free() gets called largely only by put_filp(), also in fs/file_table.c. get_empty_filp() and put_filp() get called by a variety of places, including fs/open.c, fs/pipe.c, mm/shmem.c and net/socket.c.