Tired of regularly logging into our servers to find out if anything is wrong, I wanted to know before something like a spam attack or a disk-eater gets out of hand.
Wrote some scripts to automate the process and email me some stats. Yes there are tools to do this for you, but then you have to maintain those tools, and sometimes installing them and configuring them on so many servers can be a pain. This system needs only ssh and mutt, lightweight and available in almost all Linux distributions I know.
This, it’s simply bash installed on one server, and everything is “pushed” fresh to the servers every time it is run, so updates are automatic and easily deployed.
Design
The main script checkserver.sh copies (using scp) the checkservices, checkdf, checkmailq and checkmaillog scripts and config files to the target host, runs them there and emails the output to you.
On your host servers there is nothing to install except openssh-server. Configure your password-less ssh-keys for all the servers in ~/.ssh/config on your monitoring server with the public keys in /root/.ssh/authorized_keys on the hosts. For help with that see our article: master-ssh-key
Scripts
To send notification emails you need a mutt wrapper, almost all distributions have mutt in their repos.
muttcc.sh
func_muttemail () {
#### establish current dir
APPDIR=$( cd “$( dirname “$0″ )” && pwd )
gotdate=`date +%Y-%m-%d-%H-%M`
DEBUGLOG=”$APPDIR/functions-email.log”
echo “ENTERING mutt.sh” | tee -a $DEBUGLOG#### ASSIGN VARS
mailentity=”$1″
mailsubject=”$2″
maildatafile=”$3″
mailentitycc=”$4″
echo “$mailentity $mailsubject $maildatafile” | tee -a $DEBUGFILE
#send email
echo “cc specified: $4″ | tee -a $DEBUGLOG
if [ x”${mailentitycc}” = x ]
then
#the cc is not set so do not include mecc
mutt -F $APPDIR/muttrc -s “$mailsubject” “$mailentity” < $maildatafile
echo “no cc specified: $mailentitycc” | tee -a $DEBUGLOG
else
mutt -F $APPDIR/muttrc -s “$mailsubject” -c “$mailentitycc” “$mailentity” < $maildatafile
echo “cc specified: $mailentitycc” | tee -a $DEBUGLOG
fiEX=$?
echo “$EX”
echo “##### EXITING mutt.sh #######” | tee -a $DEBUGLOG
return $EX
}
You’ll need to install mutt and have access to an smtp server. Mine is local and needs no authentication. Normally this fill will be the user’s “.muttrc” file, but in the wrapper we have specified it’s location.
muttrc
set from = “me@myserver.net”
set realname = “CHECKSERVER”
set use_envelope_from = yes
set smtp_url=smtp://mylocalmailserverhostname.net/
set ssl_starttls = no
checkservers.sh
#!/bin/bash
APPDIR=$( cd “$( dirname “$0″ )” && pwd )
. /$APPDIR/muttcc.sh
SERVERLIST=$APPDIR/server.list
MAILQTESTS=$APPDIR/checkmailq.txt
MAILLOGTESTS=$APPDIR/checkmaillog.txt
MAILQSH=$APPDIR/checkmailq.sh
MAILLOGSH=$APPDIR/checkmaillog.sh
DISKDF=$APPDIR/checkdiskdf.sh
CHECKLOG=/tmp/servercheck.logwhile IFS=”:” read servername ifmailq ifmaillog ifdiskdf serviceslist
do
echo “=========== SERVERCHECK STARTS: $servername ==============”
echo “=========== SERVERCHECK STARTS: $servername ==============” > $CHECKLOG
echo “———————- disks —————— ” >> $CHECKLOG
if [ “$ifdiskdf” == “diskdf” ]
then
scp $APPDIR/checkdiskdf.sh root@$servername:/usr/local/sbin/
ssh -n root@$servername /usr/local/sbin/checkdiskdf.sh >> $CHECKLOG
else
echo “$servername diskdf not flagged for check” >> $CHECKLOGfi
echo “———————- services —————— ” >> $CHECKLOG
echo “SERVICEFILE: $serviceslist”
case $serviceslist in
no)
echo “$servername services not flagged for check” >> $CHECKLOG
;;
*)
scp $APPDIR/checkservices-$serviceslist.txt root@$servername:/usr/local/sbin/checkservices.txt
scp $APPDIR/checkservices.sh root@$servername:/usr/local/sbin/
ssh -n root@$servername /usr/local/sbin/checkservices.sh >> $CHECKLOG
esacecho “———————- mailq —————— ” >> $CHECKLOG
if [ “$ifmailq” == “mailq” ]
then
scp $APPDIR/checkmailq.sh root@$servername:/usr/local/sbin/
scp $APPDIR/checkmailq.txt root@$servername:/usr/local/sbin/
ssh -n root@$servername /usr/local/sbin/checkmailq.sh >> $CHECKLOG
else
echo “$servername mailq not flagged for check” >> $CHECKLOG
fiecho “—————– maillog —————– ” >> $CHECKLOG
if [ “$ifmaillog” == “maillog” ]
then
scp $APPDIR/checkmaillog.sh root@$servername:/usr/local/sbin/
scp $APPDIR/checkmaillog.txt root@$servername:/usr/local/sbin/
ssh -n root@$servername /usr/local/sbin/checkmaillog.sh >> $CHECKLOG
else
echo “$servername maillog not flagged for check” >> $CHECKLOG
fi
echo “=========== SERVERCHECK ENDS: $servername ==============” >> $CHECKLOG
email_subject=”checkserver results: $servername”
echo “$email_subject”
tmp2=$(func_muttemail “myemail@address.com” “$email_subject” “$CHECKLOG” “ccemail@address.com”)
echo “=========== SERVERCHECK ENDS: $servername ==============”done < “$SERVERLIST”
checkdiskdf.sh
#!/bin/bash
/bin/df -h
exit
checkmaillog.sh
#!/bin/bash
APPDIR=$( cd “$( dirname “$0″ )” && pwd )
#greps maillotgs and mailqueue fiels for various issues
MAILLOGTESTS=$APPDIR/checkmaillog.txt
MAILRESULT=$APPDIR/checkmailresult.log
MAILRESULTREPORT=$APPDIR/checkmailresult.log
MAILRESULTTEMP=$APPDIR/checkmailresulttemp.log
rm $MAILRESULTTEMP
while IFS=”:” read testname before after
do
datum=`date “+%b %d”`
grep -A$after -B$before “$testname” /var/log/maillog | grep “$datum” >> $MAILRESULTTEMPdone < $MAILLOGTESTS
resultlength=$(cat “$MAILRESULTTEMP” | wc -l)
echo “$resultlength”
if [ “$resultlength” -gt “1” ]
then
echo “issues found”
echo “mail log errors found. lines=$resultlength.” > $MAILRESULTREPORT
cat “$MAILRESULTTEMP” >> $MAILRESULTREPORT
cat “$MAILRESULTREPORT”
else
echo “nothing to report”
fi
checkmaillog.txt
refused to talk to me:0:0
Connection timed out:0:0
Name service error for name:0:0
sender non-delivery notification:0:0
to=<noreply@:0:0
421 Too many concurrent SMTP connections:0:0
testing something:0:0
checkmailq.sh
#!/bin/bash
APPDIR=$( cd “$( dirname “$0″ )” && pwd )
#greps maillotgs and mailqueue fiels for various issuesMAILQTESTS=$APPDIR/checkmailq.txt
MAILRESULT=$APPDIR/checkmailresult.log
MAILRESULTREPORT=$APPDIR/checkmailresult.log
MAILRESULTTEMP=$APPDIR/checkmailresulttemp.log
rm $MAILRESULTTEMPwhile IFS=”:” read testname before after
do
/usr/bin/mailq | grep -A$after -B$before “$testname” >> $MAILRESULTTEMP
done < $MAILQTESTS
resultlength=$(cat “$MAILRESULTTEMP” | wc -l)
echo “$resultlength”
if [ “$resultlength” -gt “1” ]
then
echo “issues found”
echo “there is queued mail. lines=$resultlength.” > $MAILRESULTREPORT
cat “$MAILRESULTTEMP” >> $MAILRESULTREPORT
cat “$MAILRESULTREPORT”
else
echo “nothing to report”
fi
checkmailq.sh
#!/bin/bash
APPDIR=$( cd “$( dirname “$0″ )” && pwd )
#greps maillotgs and mailqueue fiels for various issuesMAILQTESTS=$APPDIR/checkmailq.txt
MAILRESULT=$APPDIR/checkmailresult.log
MAILRESULTREPORT=$APPDIR/checkmailresult.log
MAILRESULTTEMP=$APPDIR/checkmailresulttemp.log
rm $MAILRESULTTEMPwhile IFS=”:” read testname before after
do
/usr/bin/mailq | grep -A$after -B$before “$testname” >> $MAILRESULTTEMP
done < $MAILQTESTS
resultlength=$(cat “$MAILRESULTTEMP” | wc -l)
echo “$resultlength”
if [ “$resultlength” -gt “1” ]
then
echo “issues found”
echo “there is queued mail. lines=$resultlength.” > $MAILRESULTREPORT
cat “$MAILRESULTTEMP” >> $MAILRESULTREPORT
cat “$MAILRESULTREPORT”
else
echo “nothing to report”
fi
checkmailq.txt
Connection timed out:1:1
Connection refused:1:1
checkservices.sh
#!/bin/bash
#!/bin/bash
APPDIR=$( cd “$( dirname “$0″ )” && pwd )
#greps maillotgs and mailqueue fiels for various issues
SERVICETESTS=$APPDIR/checkservices.txt
RESULTREPORT=$APPDIR/checkservicesresultreport.log
RESULTTEMP=$APPDIR/checkservicesresulttemp.log
rm $RESULTREPORT
rm $RESULTTEMP
while IFS=”:” read portnumber service
do result=$( lsof -i:$portnumber | head -n2 > $RESULTTEMP)
resultlength=$(cat “$RESULTTEMP” | wc -l )
if [ “$resultlength” -gt “1” ] then
#service is running there
echo “OK: $service on $portnumber” >> $RESULTREPORT else
echo “++++++ NOK: $service on $portnumber +++++++++” >> $RESULTREPORT
fi
done < $SERVICETESTS
cat “$RESULTREPORT”
checkservices-ALL.txt
22:ssh
443:https
993:imaps
3306:mysql
389:ldap
servers.list
This is your list of servers to process.
megatron:nomailq:nomaillog:diskdf:ALL
magdelena:mailq:maillog:diskdf:ALL
In my crontab. I don’t want email noise from cron so I use /dev/null 2> &1 to null that.
30 08 * * * /usr/local/sbin/backup_app/checkservers.sh > /dev/null 2>&1
Debugging
Debug email issues with functions-email.log
For output issues log into your remote servers and run the check*.sh files to see what comes out.
Room for improvement
tmp files are used per server in the main loop, so take care not to run this script again until complete or you’ll get nonsense.