The Multi Router Traffic Grapher (MRTG) for the System Adminstrator



Installing MRTG

Red Hat Linux
Solaris

MRTG Proof of Concept

1.  Create a script to generate output in the proper format (low value high value name).

mrtg_poc.sh

    echo "10 5 Testing"

2.  Create an mrtg.cfg file

/opt/mrtg/mrtg.cfg

WorkDir: /root
Target[Server1]: `/root/mrtg_poc.sh`
Title[Server1]: Mrtg Proof of Concept
Pagetop[Server1]: Sar info from Server1
Maxbytes[Server1]: 13

3.  Execute mrtg at the command line to check for errors

/usr/local/src/mrtg-2/bin/mrtg /opt/mrtg/mrtg.cfg

3.  Bring up webpage:

file:///root//Server1.html


Example System:  Using scripts to gather %CPU and %WIO on multiple servers.

1.  Script to gather information from remote servers anc copy the data to the mrtg server.  Repeat this code for each server that you are collecting data from:

#
#       Server1
#
datadir="/tmp"
server="Server1"
cpu_data=$server"_cpu.txt"
wio_data=$server"_wio.txt"
#
#    Get cpu utilization and wio from Server1
#
rsh $server "sar 1 1"|sed 1,4d|awk '{print $5}'>$datadir/$cpu_data
rsh $server "sar 1 1"|sed 1,4d|awk '{print $4}'>$datadir/$wio_data
#
#    Write the data to Mrtg Server
rcp $datadir/$cpu_data mrtgserver:/root/$cpu_data
rcp $datadir/$wio_data mrtgserver:/root/$wio_data


2.  Schedule the statisitics script to run at the proper time intervals using crontab:

#
#   Statistics on linux server mrtgserver- Every 5 minutes
#
00,05,10,15,20,25,30,35,40,45,50,55 * * * * /opt/scripts/wte_mrtg_stats.sh

3.  mrtg.cfg file on the mrtg server to collect these statistics (each server requires these 25 lines for %cpu and %wio):

Target[server1]: `/root/server1_cpu.pl`
Colours[server1]: LIGHT BLUE#7AAFFF,BLUE#1000FF,DARK GREEN#006000,VIOLET#FF00FF
Options[server1]: gauge,noinfo, nopercent, growright, unknaszero
MaxBytes[server1_cpu]: 200
Title[server1_cpu]: Server server1 % CPU Utililization
YLegend[server1_cpu]: % CPU
ShortLegend[server1_cpu]: % CPU
Legend1[server1_cpu]: CPU Utilization
Legend2[server1_cpu]:
LegendI[server1_cpu]:
LegendO[server1_cpu]: CPU Utilization :
PageTop[server1_cpu]: <b>% CPU Utilization of Server bplex_db</b>
YSize[server1_cpu]: 100
Target[server1_wio]: `/root/server1_wio.pl`
Colours[server1_wio]: LIGHT BLUE#7AAFFF,BLUE#1000FF,DARK GREEN#006000,VIOLET#FF00FF
Options[server1_wio]: gauge,noinfo, nopercent, growright, unknaszero
MaxBytes[server1_wio]: 200
Title[server1_wio]: Server server1 % WIO
YLegend[server1_wio]: % WIO
ShortLegend[server1_wio]: % WIO
Legend1[server1_wio]: % WIO
Legend2[server1_wio]:
LegendI[server1_wio]:
LegendO[server1_wio]: % WIO :
PageTop[server1_wio]: <b>% WIO on Server server1</b>

4.  Sample files to create mrtg compatible input:

[root@lupus root]# cat server1_cpu.pl
#!/usr/bin/perl

$cpu = `cat /root/server1_cpu.txt`;
$cpu = 100-int($cpu);

print "$cpu\n";
print "$cpu\n";
print "server1\n";
print "cpu utilization\n";

[root@lupus root]# cat server1_wio.pl
#!/usr/bin/perl

$wio = `cat /root/server1_wio.txt`;
$wio = int($wio);

print "$wio\n";
print "$wio\n";
print "server1\n";
print "wio\n";

5.  Configure mrtg to run automatically in the crontab:

04,09,14,19,24,29,34,39,44,49,54,59 * * * * /usr/local/src/mrtg-2/bin/mrtg /opt/mrtg/mrtg.cfg 2> /dev/null

Tips for Monitoring Systems using scripts

Monitoring cpu utilization

1.  Create a script to get cpu utilization from a list of servers and write it to a file:

cat /opt/scripts/wte_mrtg_server_cpu.sh

datadir="/opt/mrtg/server_stats"
server_list="bplex_db scorpius bpapp_gvl gvlscsun001"
for server in $server_list
do
rsh $server "sar 1 1"|sed 1,4d|awk '{print $5}'>$datadir/$server"_cpu.txt"
done

2.  Create a script to take the cpu utilization and present it to mrtg in the proper format:

cat /opt/mrtg/bplex_db_cpu.pl

#!/usr/bin/perl
$cpu = `cat /opt/mrtg/server_stats/bplex_db_cpu.txt`;
$cpu = 100-int($cpu);

print "$cpu\n";
print "$cpu\n";
print "bplex_db\n";
print "cpu utilization\n";


3.  Configure mrtg.cfg to run the cpu utilization program

cat /opt/mrtg/mrtg.cfg

WorkDir: /var/www/html/mrtg
Target[bplex_cpu]: `/opt/mrtg/bplex_db_cpu.pl`
Colours[bplex_cpu]: LIGHT BLUE#7AAFFF,BLUE#1000FF,DARK GREEN#006000,VIOLET#FF00FF
Options[bplex_cpu]: gauge,noinfo, nopercent, growright, unknaszero
MaxBytes[bplex_cpu]: 200
Title[bplex_cpu]: Server bplex_db % CPU Utililization
YLegend[bplex_cpu]: % CPU
ShortLegend[bplex_cpu]: % CPU
Legend1[bplex_cpu]: CPU Utilization
Legend2[bplex_cpu]:
LegendI[bplex_cpu]:
LegendO[bplex_cpu]: CPU Utilization :
PageTop[bplex_cpu]: <b>% CPU Utilization of Server bplex_db</b>
YSize[bplex_cpu]: 100


4.  Schedule in crontab the collection of cpu utilization and the graphing of the data in mrtg:

00,05,10,15,20,25,30,35,40,45,50,55 * * * * /opt/scripts/wte_mrtg_server_cpu.sh 2> /dev/null
04,09,14,19,24,29,34,39,44,49,54,59 * * * * /usr/local/src/mrtg-2/bin/mrtg /opt/mrtg/mrtg.cfg 2> /dev/null


Monitoring wio


1.  Create a script to get cpu wio from a list of servers and write it to a file:

cat /opt/scripts/wte_mrtg_server_cpu.sh

datadir="/opt/mrtg/server_stats"
server_list="bplex_db scorpius bpapp_gvl gvlscsun001"
for server in $server_list
do
rsh $server "sar 1 1"|sed 1,4d|awk '{print $4}'>$datadir/$server"_wio.txt"
done


2.  Create a script to take the wio and present it to mrtg in the proper format:

cat /opt/mrtg/bplex_db_cpu.pl

#!/usr/bin/perl

$wio = `cat /opt/mrtg/server_stats/bplex_db_wio.txt`;
$wio = int($wio);

print "$wio\n";
print "$wio\n";
print "bplex_db\n";
print "wio\n";



3.  Configure mrtg.cfg to run the wio program

cat /opt/mrtg/mrtg.cfg

WorkDir: /var/www/html/mrtg
Target[bplex_wio]: `/opt/mrtg/bplex_db_wio.pl`
Colours[bplex_wio]: LIGHT BLUE#7AAFFF,BLUE#1000FF,DARK GREEN#006000,VIOLET#FF00FF
Options[bplex_wio]: gauge,noinfo, nopercent, growright, unknaszero
MaxBytes[bplex_wio]: 200
Title[bplex_wio]: Server bplex_db % WIO
YLegend[bplex_wio]: % WIO
ShortLegend[bplex_wio]: % WIO
Legend1[bplex_wio]: % WIO
Legend2[bplex_wio]:
LegendI[bplex_wio]:
LegendO[bplex_wio]: % WIO :
PageTop[bplex_wio]: <b>% WIO on Server bplex_db</b>


4.  Schedule in crontab the collection of wio and the graphing of the data in mrtg:

00,05,10,15,20,25,30,35,40,45,50,55 * * * * /opt/scripts/wte_mrtg_server_wio.sh 2> /dev/null
04,09,14,19,24,29,34,39,44,49,54,59 * * * * /usr/local/src/mrtg-2/bin/mrtg /opt/mrtg/mrtg.cfg 2> /dev/null


Monitoring tape drive utilization

HP Omniback
omniback  -> uma -ioctl /dev/picker < /tmp/uma_test.txt|grep -c Full    <-- on server that is Cell Manager

Veritas Netbackup
/usr/openv/volmgr/bin/vmoprcmd|grep dlt|grep -v -|wc -l     <-- on server that is Media Master

Monitoring Netbackup scratch tapes

1.  Create a script to get the number of scratch tapes from the netbackup server and write it to a file:

cat wte_mrtg_backup.sh

#       Get the number of scratch tapes from the netbackup server:
server="scorpius"
datadir="/opt/mrtg/server_stats"
datafile=$server"_scratch.txt"
rsh $server "/usr/openv/volmgr/bin/vmquery -pn Scratch"|grep -c "robot slot" > $datadir/$datafile


2.  Create a script to take the number of scratch tapes and present it to mrtg in the proper format:

cat scorpius_scratchtape.pl

#!/usr/bin/perl

$scratchtape = `cat /opt/mrtg/server_stats/scorpius_scratch.txt`;
chomp $scratchtape;
print "$scratchtape\n";
print "$scratchtape\n";
print "scorpius\n";
print "scratchtape count\n";


3.  Configure mrtg.cfg to run the scratch tape program

Target[scorpius_scratch]: `/opt/mrtg/scorpius_scratchtape.pl`
Colours[scorpius_scratch]: LIGHT BLUE#7AAFFF,BLUE#1000FF,DARK GREEN#006000,VIOLET#FF00FF
Options[scorpius_scratch]: gauge,noinfo, nopercent, growright, unknaszero
MaxBytes[scorpius_scratch]: 200
Title[scorpius_scratch]: Server scorpius Scratch Tapes Available
YLegend[scorpius_scratch]: Scratch Tapes
ShortLegend[scorpius_scratch]: Scratch
Legend1[scorpius_scratch]: Scratch Tapes
Legend2[scorpius_scratch]:
LegendI[scorpius_scratch]:
LegendO[scorpius_scratch]: Scratch Tapes
PageTop[scorpius_scratch]: <b>Number of Scratch Tapes Available on Server scorpius</b>


4.  Schedule in crontab the collection of wio and the graphing of the data in mrtg:

02,07,12,17,22,27,32,37,42,47,52,57 * * * * /opt/scripts/wte_mrtg_backup.sh 2> /dev/null

Monitoring HP Data Protector Scratch Tapes (Free Pool)

omnimm -list_pool "Free DLT"|grep -c <Library Name>   <-- ie 10/180

Monitoring HP Data Protector Tapes marked as Poor

omnimm -list_pool |sed 1,3d|awk '{print $2}'|xargs -i omnimm -list_pool {}|sed 1,3d|grep Poor |grep -c 10/180


Monitoring sar on a linux box:

1.  Capture the data on the linux box via cron every 5 minutes and rcp it to the monitoring server storage location:

cron job:

0,5,10,15,20,25,30,35,40,45,50,55 * * * * /opt/scripts/nusales_cpu.sh

script:

# cat nusales_cpu.sh
#/usr/bin/sar 1 1|/usr/bin/tail -2|/usr/bin/head -1|/bin/awk '{print $7}' > /tmp/nusales2_cpu.txt
/usr/bin/sar 1 1  |/usr/bin/tail -2 | /usr/bin/head -1 | cut -c55-80 > /tmp/nusales2_cpu.txt
/usr/bin/rcp /tmp/nusales2_cpu.txt castor:/tmp/nusales2_cpu.txt


2.  Copy the data from the storage location on the monitoring server to the mrtg server:

cronjob:
00,05,10,15,20,25,30,35,40,45,50,55 * * * * /opt/scripts/wte_mrtg_stats.sh

script:

#
#       nusales2
#
datadir="/tmp"
nusales2_cpu="nusales2_cpu.txt"
#
#    Write the data to lupus
rcp $datadir/$nusales2_cpu lupus:/root/$nusales2_cpu


3.   On the mrtg box configure the mrtg.cfg and associated scripts:

mrtg.cfg:

Target[nusales2_cpu]: `/root/nusales2_cpu.pl`
Colours[nusales2_cpu]: LIGHT BLUE#7AAFFF,BLUE#1000FF,DARK GREEN#006000,VIOLET#FF00FF
Options[nusales2_cpu]: gauge,noinfo, nopercent, growright, unknaszero
MaxBytes[nusales2_cpu]: 200
Title[nusales2_cpu]: Server nusales2 % CPU Utililization
YLegend[nusales2_cpu]: % CPU
ShortLegend[nusales2_cpu]: % CPU
Legend1[nusales2_cpu]: CPU Utilization
Legend2[nusales2_cpu]:
LegendI[nusales2_cpu]:
LegendO[nusales2_cpu]: CPU Utilization :
PageTop[nusales2_cpu]: <b>% CPU Utilization of Server nusales2</b>
YSize[nusales2_cpu]: 100


nusales2_cpu.pl:

#!/usr/bin/perl


$cpu = `cat /root/nusales2_cpu.txt`;
$cpu = 100-int($cpu);

print "$cpu\n";
print "$cpu\n";
print "nusales2\n";
print "cpu utilization\n";


Monitoring memory on a Linux box:

1.  On the target box:

#  MRTG Monitoring scripts
00,05,10,15,20,25,30,35,40,45,50,55 * * * * /opt/scripts/serverx_mem.sh

# cat serverx_mem.sh
free -m |grep Mem|awk '{print $4}' > /tmp/serverx_mem.txt
rcp /tmp/serverx_mem.txt monserver:/tmp/serverx_mem.txt

2.  On the monitoring/storage box:

#
#       Serverx
#
datadir="/tmp"
serverx_mem="serverx_mem.txt"
#
#    Write the data to mrtg server
rcp $datadir/$serverx_mem mrtgserver:/root/$serverx_mem


3.  On the mrtg server:

create the script to output memory statistics to mrtg:

# cat serverx_mem.pl
#!/usr/bin/perl


$mem = `cat /root/serverx_mem.txt`;
$mem = int(($mem/1511) * 100);           <-- Measurements are setup as a %

print "$mem\n";
print "$mem\n";
print "serverx\n";
print "mem available\n";



update the mrtg.cfg file for the new server information