Building and Configuring a Central Logging Server with syslog-ng
Introduction
This article describes the process of replacing the venerable but limited syslog daemon with a versatile, flexible and customisable replacement, syslog-ng. We will be using our syslog-ng enabled host as a central logging server, so we'll use LVM to create a log volume that can easily be expanded as disks are added to the system.
Preparation - Configure Log Volume Using LVM
I am using a Red Hat Enterprise Linux AS 4 Update 2 host for my logging server. This has four SCSI disks attached, one 10Gb (for the / filesystem, plus /boot, /var, /var/log, swap, /tmp, /home). Three 5Gb disks are attached that will be used to create the volume upon which our logs will be stored. For the purpose of illustration, I will use two disks initially, and then extend the volume to encompass the third disk.
Now, we'll join the build process as I've just powered down my logging host, attached my three 5Gb disks and powered the server back on. It is time to create our log volume.
First, prepare the disks that will form our log volume with fdisk. I will just use two disks initially, then extend the volume onto the third disk to illustrate the process. As you can see, we set the partition type to 8e, for Linux LVM. I'll show all command input, and will also show command output where it's useful/unique/interesting.
# fdisk /dev/sdb
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-652, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-652, default 652):
Using default value 652
Command (m for help): t
Selected partition 1
Hex code (type L to list codes): 8e
Changed system type of partition 1 to 8e (Linux LVM)
Command (m for help): p
Disk /dev/sdb: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 652 5237158+ 8e Linux LVM
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
We do the same for /dev/sdc
# fdisk /dev/sdc # follow same process as per /dev/sdb
As this is the first time we're using LVM on the host, we'll run vgscan to initialise the LVM facility
# vgscan -v
Creating directory "/var/lock/lvm"
Wiping cache of LVM-capable devices
Wiping internal VG cache
Reading all physical volumes. This may take a while...
Finding all volume groups
No volume groups found
Now, we can designate our physical disk partitions as physical volumes, and make them available for LVM use.
# pvcreate /dev/sdb1 /dev/sdc1
Physical volume "/dev/sdb1" successfully created
Physical volume "/dev/sdc1" successfully created
We can now create our volume group, which will will define as vg1
# vgcreate vg1 /dev/sdb1 /dev/sdc1
Volume group "vg1" successfully created
I will now create a 9Gb logical volume on the volume group
# lvcreate -L 9G -n "log_lv" vg1
Logical volume "log_lv" created
The next step is to create a filesystem on our logical volume. The standard mkfs tools can be used for this.
# mkfs.ext2 -j /dev/vg1/log_lv
mke2fs 1.35 (28-Feb-2004)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
1179648 inodes, 2359296 blocks
117964 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2415919104
72 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 20 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
Now we can test the volume by mounting it
# mount -t ext3 /dev/vg1/log_lv /mnt
# mount | grep log_lv
/dev/mapper/vg1-log_lv on /mnt type ext3 (rw)
# umount /mnt
All good. Now, let's extend our volume across the third disk. First, we must create a partition using fdisk
# fdisk /dev/sdd # follow same process as per /dev/sdb
Next, use pvcreate to designate the partition for use by LVM, and vgextend to add the partiton to our vg1 volume group
# pvcreate /dev/sdd1
Physical volume "/dev/sdd1" successfully created
# vgextend vg1 /dev/sdd1
Volume group "vg1" successfully extended
Let's use vgdisplay to check that things are looking good for our vg1 volume group
# vgdisplay vg1
--- Volume group ---
VG Name vg1
System ID
Format lvm2
Metadata Areas 3
Metadata Sequence No 3
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 0
Max PV 0
Cur PV 3
Act PV 3
VG Size 14.98 GB
PE Size 4.00 MB
Total PE 3834
Alloc PE / Size 2304 / 9.00 GB
Free PE / Size 1530 / 5.98 GB
VG UUID 8WqAJ1-d7aM-Vj6x-m6Ap-VwRX-zlSu-lN4LQ9
Now we can resize our logical volume, log_lv.
# lvextend -L 14G /dev/vg1/log_lv
Extending logical volume log_lv to 14.00 GB
Logical volume log_lv successfully resized
Next we check the filesystem with fsck, and then resize our ext3 filesystem across this newly allocated space.
# e2fsck -f /dev/vg1/log_lv
e2fsck 1.35 (28-Feb-2004)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/vg1/log_lv: 11/1179648 files (9.1% non-contiguous), 50409/2359296 blocks
# resize2fs /dev/vg1/log_lv 14G
resize2fs 1.35 (28-Feb-2004)
Resizing the filesystem on /dev/vg1/log_lv to 3670016 (4k) blocks.
The filesystem on /dev/vg1/log_lv is now 3670016 blocks long.
Now, let's mount the filesystem and ensure that all is well.
# mount -t ext3 /dev/vg1/log_lv /mnt
# mount | grep log_lv
/dev/mapper/vg1-log_lv on /mnt type ext3 (rw)
# df -h /mnt
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg1-log_lv
14G 55M 14G 1% /mnt
# umount /mnt
We now have a 14G volume ready to use as our log volume. I'll now add a permanent mountpoint (/var/syslog-ng) and add an entry to /etc/fstab so that the filesystem is mounted persistently
# mkdir /var/syslog-ng
# vi /etc/fstab
# grep syslog-ng /etc/fstab
/dev/vg1/log_lv /var/syslog-ng ext3 defaults 1 2
# mount -a
# mount | grep syslog-ng
/dev/mapper/vg1-log_lv on /var/syslog-ng type ext3 (rw)
Downloading the Required Packages
Now that our log volume is ready for use, we can start downloading the required packages for our logging server. I am compiling syslog-ng from source (http://www.balabit.com/downloads/syslog-ng/1.6/src/syslog-ng-1.6.11.tar.gz) and this has a prerequisite dependency on libol, so I'll download that too (http://www.balabit.com/downloads/libol/0.3/libol-0.3.18.tar.gz). For viewing the (eventual) logfiles, I recommend multitail - an amazingly feature rich tail replacement and colorizer (http://dag.wieers.com/packages/multitail/multitail-4.0.5-1.el4.rf.i386.rpm)
# mkdir -p /usr/local/src/logging
# cd /usr/local/src/logging
# wget http://www.balabit.com/downloads/syslog-ng/1.6/src/syslog-ng-1.6.11.tar.gz
# wget http://www.balabit.com/downloads/libol/0.3/libol-0.3.18.tar.gz
# wget http://dag.wieers.com/packages/multitail/multitail-4.0.5-1.el4.rf.i386.rpm
The Build Process
Before we can build syslog-ng, we must first build libol on which syslog-ng is dependent. The usual build process applies...
# cd /usr/local/src/logging
# gzip -dc ./libol-0.3.18.tar.gz | tar xf -
# cd libol-0.3.18
# ./configure
# make
# make install
If no errors are encountered, we can proceed to building syslog-ng. I'm happy with everything ending up in /usr/local so no need to modify --prefix during configure. Make sure you have flex installed too... configure will complete, but the make will barf without it.
# cd ..
# gzip -dc ./syslog-ng-1.6.11.tar.gz | tar xf -
# cd syslog-ng-1.6.11
# ./configure
# make
# make install
While we're at it, I'll now install the multitail RPM.
# rpm -Uvh ./multitail-4.0.5-1.el4.rf.i386.rpm
Completing the syslog-ng Installation and Configuration
Now that we have syslog-ng installed (our binary is at /usr/local/sbin/syslog-ng) we need to do a few things before we can start using it. First, we must install a start/stop script under /etc/init.d for syslog-ng. A script comes with the syslog-ng bundle that will suffice for our purposes.
# cd /usr/local/src/logging/syslog-ng-1.6.11/contrib/rhel-packaging/
# cp ./syslog-ng.init /etc/init.d/
# mv /etc/init.d/syslog-ng.init /etc/init.d/syslog-ng
# chown root:root /etc/init.d/syslog-ng
# chmod 744 /etc/init.d/syslog-ng
Before we can chkconfig this service, we must first modify the start/stop script slightly to point to the correct syslog-ng binary.
# vi /etc/init.d/syslog-ng
# grep "^binary" /etc/init.d/syslog-ng
binary="/usr/local/sbin/syslog-ng"
OK, next, we need to create our configuration file at /usr/local/etc/syslog-ng/syslog-ng.conf
Much of the configuration for this file is straightforward, and a detailed discussion is beyond the scope of this article - man syslog-ng.conf will give you the required information. I have created my own configuration file to mimick the functionality in the standard RHEL4 /etc/syslog.conf. As well as this, I have tailored the file to capture remote (and local) messages and direct them to the appropriate file under /var/syslog-ng.
# cd /usr/local/etc
# mkdir syslog-ng
# cd syslog-ng
# cat > ./syslog-ng.conf
# mphost04 - syslog-ng Configuration File
# Global options
options {
sync (0);
time_reopen (10);
log_fifo_size (1000);
long_hostnames (off);
use_dns (no);
use_fqdn (no);
create_dirs (no);
keep_hostname (yes);
stats (3600);
};
# Define local system
source s_sys {
pipe ( "/proc/kmsg" log_prefix ( "kernel: " ) );
unix-stream ( "/dev/log" );
internal();
};
# Define network
source s_net { udp (); };
# The following logging has been designed to roughly mimick RHEL4s standard
# /etc/syslog.conf. We are logging *everything* to the /var/syslog-ng filesystem
# from this system, but still want to maintain traditional local logs.
# Define local destinations
destination d_cons { file ( "/dev/console" ); };
destination d_mesg { file ( "/var/log/messages" ); };
destination d_auth { file ( "/var/log/secure" ); };
destination d_mail { file ( "/var/log/maillog" ); };
destination d_spol { file ( "/var/log/spooler" ); };
destination d_boot { file ( "/var/log/boot.log" ); };
destination d_cron { file ( "/var/log/cron" ); };
destination d_mlal { usertty ( "*" ); };
destination d_kern { file ( "/var/log/kern" ); };
# Define local filters
filter f_mesg {
level ( info..emerg ) and not ( facility ( mail ) or facility ( authpriv ) or facility ( cron ) );
};
filter f_auth { facility ( authpriv ); };
filter f_mail { facility ( mail ); };
filter f_cron { facility ( cron ); };
filter f_emrg { level ( emerg ); };
filter f_spol { facility ( uucp ) or ( facility ( news ) and level ( crit ) ); };
filter f_boot { facility ( local7 ); };
# Define local logging
log { source ( s_sys ); filter ( f_mesg ); destination ( d_mesg ); };
log { source ( s_sys ); filter ( f_auth ); destination ( d_auth ); };
log { source ( s_sys ); filter ( f_mail ); destination ( d_mail ); };
log { source ( s_sys ); filter ( f_cron ); destination ( d_cron ); };
log { source ( s_sys ); filter ( f_emrg ); destination ( d_mlal ); };
log { source ( s_sys ); filter ( f_spol ); destination ( d_spol ); };
log { source ( s_sys ); filter ( f_boot ); destination ( d_boot ); };
# Define remote logging destination - we log to our /var/syslog-ng filesystem
# and store the logs by hostname first, and then by date. We create_dirs as required
# and ensure that all files are chown root:root and chmod 600. Any dirs created are
# chmod 700. We also define a template here which will dictate the format of each log
# entry
destination d_remote_logs {
file ( "/var/syslog-ng/$FULLHOST/$YEAR/$MONTH/$DAY/$FULLHOST-$YEAR-$MONTH-$DAY.log"
owner( root ) group( root ) perm ( 0600 ) dir_perm( 0700 ) create_dirs ( yes )
template ( "$DATE $FULLHOST $PROGRAM $TAG [$FACILITY.$LEVEL] $MESSAGE\n" ) );
};
# Log to remote destination for local and incoming remote logs
log { source ( s_net ); destination ( d_remote_logs ); };
log { source ( s_sys ); destination ( d_remote_logs ); };
^D
# chmod 400 ./syslog-ng.conf
OK, we're almost there. Next, we need to stop syslogd and chkconfig it to disable the service. Then we can chkconfig syslog-ng and start it up!
# service syslog stop
Shutting down kernel logger: [ OK ]
Shutting down system logger: [ OK ]
# chkconfig --level 0123456 syslog off
# chkconfig --add syslog-ng
# chkconfig --level 016 syslog-ng off
# chkconfig --level 2345 syslog-ng on
# chkconfig --list syslog-ng
syslog-ng 0:off 1:off 2:on 3:on 4:on 5:on 6:off
# service syslog-ng start
Starting syslog-ng: [ OK ]
We now have syslog-ng running! Let's run logger, and check that the message appears in the traditional /var/log/messages file, as well as our new logfile under /var/syslog-ng/$HOSTNAME/$YEAR/$MONTH/$DAY/
# logger -p user.info "Testing syslog-ng"
# tail -1 /var/log/messages
Jun 18 21:35:47 mphost04 kevin: Testing syslog-ng
# tail -1 /var/syslog-ng/`hostname | cut -d. -f1`/`date +%Y`/`date +%m`/`date +%d`/*.log
Jun 18 21:35:47 mphost04 kevin 0e [user.info] kevin: Testing syslog-ng
I'm happy with that, so after a few more tests with logger, remove the old syslog service from chkconfig.
# chkconfig --del syslog
Getting Remote Hosts to log to our Logging Server
Let's get a couple of our hosts logging to our new logging server. I'll show two ways of doing this (both using /etc/syslog.conf on "traditional" syslogd-based boxes ) - one on another RHEL4 host, the other Solaris 10.
rhel4# vi /etc/syslog.conf
rhel4# sed -n '$p' /etc/syslog.conf
*.* @mphost04
rhel4# service syslog restart
Of course, ensure that mphost04 (our logging server) is in /etc/hosts or DNS, otherwise just specify the IP address.
On our Solaris 10 box ( which is actually a sparse root zone, but that's a story for another article ;-) ), something along the lines of the following does the trick
sol10# vi /etc/syslog.conf
sol10# sed -n '$p' /etc/syslog.conf
*.debug @mphost04
sol10# svcadm refresh system-log
We should now see the logs pouring in from our remote hosts!
Using multitail to View Log Output
multitail has a *plethora* of options available. Take a look at man multitail to get an idea... There are plenty of examples of usage on the multitail homepage (http://www.vanheusden.com/multitail), but I like to have a terminal window open at all times, tailing the files and using the default syslog colour scheme.
# multitail -CS syslog -d --no-repeat --mergeall /var/syslog-ng/*/`date +%Y`/`date +%m`/`date +%d`/*
A sample of the output from multitail can be seen here.
Shortcomings
One problem with this is, if at the start of the day you fire up multitail, if a particular host has not yet logged a message for a particular day, it's directory will not exist within the /var/syslog-ng filesystem when you initiate the tail, as the wildcards are expanded by the shell before multitail is invoked. Therefore the easiest workaround for this is to create a simple cronjob that calls logger on each host to generate the log file each morning - something like
1 0 * * * /usr/bin/logger -p user.info "Initialising logfile..."
This will ensure that when you fire up multitail at the start of the day, all relevant log files are already created.
Conclusion
This has just *touched* the surface of what syslog-ng can do, but has introduced a few pther things along the way, including LVM and the very cool multitail package. Read the syslog-ng manual pages, and have a play with the available functionality. You can use FIFOs to match information in other log files, for example, and redirect that information elsewhere (or mail it, etc). Now is a good time to implement NTP on your network too if you're not already using it, otherwise your timestamps within the logfiles will be out of whack.
Cheers
Kevin Waldron
kevin@zazzybob.com
Disclaimer! - This article is provided for guidance only, and does not replace the relevant official documentation and manuals. I will not be held liable for any hosed systems and/or data.
