As a server setup gets more complex, it is possible that the default syslog (
sysklogd) lacks the necessary finely grained filtering that may be desired. Common solutions to this are using
rsyslog. I have opted to go with the latter and found it to, so far, be an excellent replacement.
Update: The latest version of Amazon’s Linux AMI uses rsyslog by default.
On Amazon’s Linux AMI,
rsyslog (v4.6.2-1.6) is available in the
amzn repository and can be installed via yum. At first glance, however, rsyslog appears to conflict with sysklog (specifically, the file
/etc/logrotate.d/syslog). Attempting to uninstall sysklog is not an option, as it appears that pretty much everything depends on it). Renaming the conflicing file, of course, does not resolve the dependency.
The solution is to use
yum shell to install rsyslog while simultaneously removing sysklog – the dependencies will be resolved without issue.
At the yum shell prompt, enter (wait for each command to execute before continuing):
- install rsyslog
- remove syslog
The overall process should resemble the following:
# yum shell
Loaded plugins: fastestmirror, security Setting up Yum Shell > install rsyslog Loading mirror speeds from cached hostfile amzn | 2.1 kB 00:00 Setting up Install Process > remove sysklogd Setting up Remove Process > run --> Running transaction check ---> Package rsyslog.i686 0:4.6.2-1.6.amzn1 set to be updated ---> Package sysklogd.i386 0:1.4.1-46.8.amzn1 set to be erased --> Finished Dependency Resolution =================================================================================================== Package Arch Version Repository Size =================================================================================================== Installing: rsyslog i686 4.6.2-1.6.amzn1 amzn 427 k Removing: sysklogd i386 1.4.1-46.8.amzn1 installed 118 k Transaction Summary =================================================================================================== Install 1 Package(s) Upgrade 0 Package(s) Remove 1 Package(s) Reinstall 0 Package(s) Downgrade 0 Package(s) Total size: 427 k Is this ok [y/N]: y Downloading Packages: Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Installing : rsyslog-4.6.2-1.6.amzn1.i686 1/2 Erasing : sysklogd-1.4.1-46.8.amzn1.i386 2/2 warning: /etc/syslog.conf saved as /etc/syslog.conf.rpmsave Removed: sysklogd.i386 0:1.4.1-46.8.amzn1 Installed: rsyslog.i686 0:4.6.2-1.6.amzn1 Finished Transaction > exit Leaving Shell
You should note that both the install and remove were processed together, and the old syslog.conf was backed up. (Additionally, the conflicting file /etc/logrotate.d/syslog is also backed up and a new (slightly different) one created).
Rsyslogd is a drop in replacement for syslogd – it starts with the same default set of rules (any custom selectors you added to syslog.conf are not copied over). It should work without any setup on a system which doesn’t have custom syslog rules setup. For systems with custom rules, you will have to manually copy the rules to the new conf file (most should not need modification). By default, rsyslogd is not started, although it is setup to start at boot. To start rsyslogd, run:
service rsyslog start
You can view startup settings for rsyslog by running:
chkconfig --list rsyslog
Which should return something similar to the following (default):
rsyslog 0:off 1:off 2:on 3:on 4:on 5:on 6:off
rsyslogd does not use the same configuration file as sysklogd, although the format shares many similarities. The file used by rsyslogd is
/etc/rsyslog.conf (while the one used by sysklogd was /etc/syslog.conf)
(It is, perhaps, worth noting, however, that (at least on Amazon’s Linux) rsyslogd’s pid file is
/var/run/syslogd.pid not /var/run/rsyslogd.pid).
As a quick example of one of the available features of rsyslogd that is not found in sysklogd, consider the following scenario:
I have suhosin installed on my server, and some scripts trigger it – however, it sends messages at a debug level, and so do other scripts. This results in suhosin notifications being logged to /var/log/messages as well as a custom log file I had previously setup.
With rsyslog, it is possible to look for a particular part of the message and filter using that, as well as to prevent subsequent selectors from being triggered.
A sample suhosin notification looks like the following:
Feb 12 15:14:01 servername suhosin: ALERT - script tried to increase memory_limit to 268435456 bytes which is above the allowed value (attacker 'xx.xxx.xx.xxx', file '/path/to/file/test.php', line 10)
While the message may change depending on the event, the program name (suhosin) remains constant and can be used to filter the message.
To setup such a filter, add the following to /etc/rsyslog.conf, at the top of the rules section (before any other rules).
:programname, contains, "suhosin" /var/log/suhosin.log & ~
This will check the ‘programname’ to see if it ‘contains’ the text ‘suhosin’, and if it does, will redirect the message to the specified log file. The next line, discards the message, so that no further rules are triggered
Oddity with SpamAssassin
After installing rsyslog, I noticed that SpamAssassin would throw the following warning when started:
warn: logger: failed to add syslog method: logger: syslog initialization failed
As such, sa-update, which restarts the service, would generate the same error (and cron would mail it out). To resolve this, modify
/etc/sysconfig/spamassassin (Amazon’s Linux, RHEL/CentOS derived) appending
SPAMDOPTIONS. The final result should resemble the following:
SPAMDOPTIONS="-d -c -m5 -H --syslog-socket=native"
One of the problems I have found with this setup is that the process appears to die periodically as per the status below:
#service rsyslog status rsyslogd dead but pid file exists
Further investigation into the problem reveals that when rsyslog receives the HUP signal, it will attempt a complete restart (however, it appears to not quite succeed in this case). The source of the HUP signal appears to be logrotate.
This result can be verified by running:
#kill -HUP `cat /var/run/syslogd.pid` #service rsyslog status
The status message is the same as the one above.
From this state, rsyslog can be directly started (without even deleting the PID file) via:
#service rsyslog start
While an upgrade of rsyslog may resolve this issue (I haven’t tried), a simpler solution appears to be a minor modification to the configuration:
Edit rsyslog.conf, add the following at the end of the Global Directives section (just before the Rules section):
Sending the HUP signal no longer results in the termination of the process making rsyslog considerably more usable.
http://www.rsyslog.com/doc/property_replacer.html (list of available properties to filter on)
http://www.rsyslog.com/doc/rsyslog_conf_filter.html (configuration options for filtering)
http://blog.gerhards.net/2008/10/new-rsyslog-hup-processing.html (rsyslog HUP processing)