Bash Scripting for System Monitoring and Alerts

System monitoring is a crucial task for maintaining the health and performance of servers. For Unix-based systems, Bash scripting offers a powerful tool to automate monitoring tasks, such as tracking CPU and memory usage, and sending alerts when these metrics exceed predefined thresholds. This article walks you through creating a Bash script for system monitoring and alerts, complete with working code examples and explanations.

1. Understanding System Monitoring Metrics

Before diving into scripting, it’s important to understand the metrics we are monitoring. Two of the most critical metrics are CPU usage and memory usage. High CPU usage over prolonged periods can indicate a bottleneck or a runaway process, while high memory usage might lead to system crashes or slowdowns.

We will use utilities like top and free to gather these metrics. These tools are available in most Unix-like systems and provide comprehensive views of system performance.

2. Gathering System Metrics with Bash

Let’s write a function in Bash to retrieve CPU and memory usage:

get_cpu_usage() {
   mpstat | awk '$12 ~ /[0-9.]+/ { print 100 - $13 }'
}

get_memory_usage() {
   free | awk '/Mem:/ { print $3/$2 * 100.0 }'
}

In these functions, mpstat provides detailed CPU usage statistics, and we calculate CPU usage by subtracting idle percentage from 100. Similarly, free is used to analyze memory usage, and the percentage is computed by dividing used memory by the total memory.

3. Creating the Alert System

Next, we need to set up conditions that will trigger alerts. Alerts can be sent via email—here’s how we can implement this using mail (ensure your system has a mail client configured):

THRESHOLD_CPU=75
THRESHOLD_MEM=80

alert_if_exceeds_threshold() {
   CPU_USAGE=$(get_cpu_usage)
   MEM_USAGE=$(get_memory_usage)

   if (( $(echo "$CPU_USAGE > $THRESHOLD_CPU" | bc -l) )); then
      echo "CPU usage is at ${CPU_USAGE}%" | mail -s "CPU Alert!" user@example.com
   fi

   if (( $(echo "$MEM_USAGE > $THRESHOLD_MEM" | bc -l) )); then
      echo "Memory usage is at ${MEM_USAGE}%" | mail -s "Memory Alert!" user@example.com
   fi
}

This function checks if the current CPU or memory usage exceeds the specified thresholds. If so, an email is sent using the mail command, notifying the user of the breach.

4. Automating the Monitoring Script

To make this script effective, it needs to run at regular intervals. One way to achieve this is by using cron jobs:

# Open the cron tab
crontab -e
# Add the following line to run the script every 5 minutes
*/5 * * * * /path/to/your/script.sh

This simple cron job runs the script every five minutes, continually monitoring the system and sending alerts as needed.

5. Enhancements and Optimization Tips

While the basic setup can effectively monitor your system, consider adding features such as logging to file, more granular metrics, or alert aggregation to avoid spamming.

For instance, adding logging can help with diagnostics:

echo "$(date): CPU ${CPU_USAGE}%, MEM ${MEM_USAGE}%" >> /var/log/sys_monitor.log

Use sleep and backoff strategies in your script to handle high alert situations gracefully and avoid sending too many emails in a short time.

Finally, always verify your alerting mechanisms by testing various threshold scenarios in a controlled environment to ensure proper functionality.

Useful links: