Tips for Log File Analysis
What to monitor
- Activity Logging - what/who
- Software Monitoring - are software jobs running properly - on
time, without failure, completing?
- Troubleshooting System Failures - specific error conditions
- Track usage of resources
Distributed Logging vs Centralized Loghost
Local Logging
- Configure local logging AND centralized logging for redundancy
Centralized Logging
- Store remote logs on a easy to access server for faster response
- Access to logs even when the target system is down
- Can help identify the cause of problems that cause server reboot
during startup - avoid extra reboots
- Hackers can't easily remove their tracks from the log files if
they are not stored locally - compare centralized logs with local
logs
- Easier correlation of unusual entries across servers
- Easier to restore a particular log from tape if all logs are from
a centralized location, rather than distributed
Types of log monitoring
- Search for patterns - search the log files for a specific pattern
- Correlation - set up rules about what should be happening - ie
whether a schedule job is running at the proper time, whether it
completed with in the expected time limits, etc.
Syslog
syslogd
- /dev/log --> device socket that receives log messages on local
machine
- /dev/klog --> device socket that receives kernel messages
- UDP Port 514 --> port used to receive log messages from
other machines
/etc/syslog.conf
- Must use tabs, not spaces
/var/adm/*
/var/log/*
Simple Watcher (Swatch)
man swatch
swatch --help
swatch --version
Swatch Configuration file (.swatchrc)
#
# Swatch configuration file
#
watchfor /invalid/
echo
throttle 30:00
watchfor /[Ff]ailed/
echo
watchfor /but got/
echo
watchfor /[Ee]rror/
echo
watchfor /Real domain name/
echo
throttle 30:00
watchfor /[Tt]imeout/
echo
Using swatch to examine files
swatch --examine=/var/log/<logfile>
Command line script to examine multiple files:
for file in `ls cron*`
do
echo "Start examination of: $file"
swatch --examine $file
echo " "
echo " "
done