Squid: Increasing Available File Descriptors
Introduction
As a site grows, so do the number of users accessing the internet. A Squid proxy server that was handling requests quite well a few months ago can easily begin having problems once more and more requests start hitting it. Compound this with a flaky switch or router upstream, and you could be asking for trouble. Why?
Each process running on a Unix system will have a maximum number of file descriptors associated with it. On many modern systems, this is (by default) 1024. Whilst this is fine for most things, some applications (particularly databases and in our case a proxy server) can have issues and run out of file descriptors. This causes poor application performance, or total application failure.
Symptoms
The first symptom of the Squid application running out of file descriptors will be messages like these in your cache.log file:
WARNING! Your cache is running out of filedescriptors
The next symptom will be frustrated users logging tickets saying that pages are taking ages to load, if they even load at all.
Chances are, if you perform an lsof -u squid | wc -l, you'll see the number of file handles in use nearing 1024....
Treatment
When you first download and ./configure Squid, the default per-process file descriptor (hereafter referred to as FD to save myself some typing) limit will be taken as the Operating System default. For example, on a Red Hat Enterprise Linux AS 4 box....
# grep '__FD' /usr/include/bits/typesizes.h
#define __FD_SETSIZE 1024
This limit will work reasonably well for a site with even a few hundred simultaneous Internet users. When you start Squid, you'll see a message in cache.log stating:
With 1024 file descriptors available
As our site is growing, let's increase the FD limit to 4096. This should be ample and accomodate further increased utilisation of our proxy servers.
Backup Squid
We're going to recompile Squid from scratch. So first of all, I want to backup the entire /usr/local/squid directory structure, and also take a seperate backup of /usr/local/squid/etc/squid.conf for easy restoration after the recompilation.
# cd /usr/local/squid
# mkdir ../squid_archive
# tar czf ../squid_archive/squid_backup.tar.gz ./*
# cp -p ./etc/squid.conf ../squid_archive
We'll now blow the original Squid installation away
# pwd
/usr/local/squid
# cd ..
# rm -rf ./squid
Rebuild Squid
We've already downloaded our source to /usr/local/src/squid, so let's unpack:
# pwd
/usr/local/src/squid
# ls
squid-2.6.STABLE3.tar.gz
# tar xzf ./squid-2.6.STABLE3.tar.gz
# cd squid-2.6.STABLE3.tar.gz
Of course, we've read the README and INSTALL documents at this point, and we're ready to proceed. Now comes the important part, ensuring that the --with-maxfd option is used when we run ./configure, and given the appropriate value for the number of FDs we want available to Squid:
# ./configure --prefix=/usr/local/squid --with-maxfd=4096
You'll see the usual tests fly past, but take note of the following:
checking Default FD_SETSIZE value... 1024
Maximum filedescriptors set to 4096
Perfect, the default FD limit has been overridden. Once ./configure has finished, we can make and make install
# make && make install
Final Steps
Now we're almost ready to go. A lot of our configuration (init scripts, squid user, etc.) already exist from our previous Squid installation, so we don't need to worry about that. One thing we do need to do is recreate our cache (in reality, you'd have a seperate volume mounted on /usr/local/squid/cache), and I recursively chown the entire /usr/local/squid hierarchy:
# mkdir /usr/local/squid/cache
# chown -R squid:squid /usr/local/squid
Now we can restore our configuration file:
# cd /usr/local/squid/etc
# cp -p ./squid.conf ./squid_orig.conf
# cp -p ../../squid_archive/squid.conf .
Let's recreate our cache
# /usr/local/squid/sbin/squid -z
Now, for a very important step. Before starting Squid, you need to set the per-process limit at a shell level using ulimit, otherwise Squid will still not be able to take advantage of it's raised FD limit. Once you've verified that all is working, add the ulimit command to /etc/init.d/squid....
# ulimit -HSn 4096
Now we can start Squid, and check that our cache.log file reflects the raised FD limit:
# /usr/local/squid/sbin/squid
# grep '4096' /usr/local/squid/var/logs/cache.log
2006/09/05 23:35:46| With 4096 file descriptors available
Excellent - that's it!
Cheers
Kevin Waldron
kevin@zazzybob.com
Disclaimer! - This article is provided for guidance only, and does not replace the relevant official documentation and manuals. I will not be held liable for any hosed systems and/or data.
