Complete Guide to a Complete Linux Server

34. Install System Health and Security Probe

by Double Bastion - Updated November 28, 2024

System Health and Security Probe is a program that runs periodically and sends a report email to the administrator if it detects at least one of the following problems:

  • Any service running on the host server is in failed state;
  • The periodic ClamAV scan has detected any viruses in the mail directories or in the Nextcloud upload directories, or the logs show recent virus detections in the incoming emails or in the files uploaded to Nextcloud;
  • Any new IP address has been banned during the last run cycle interval due to repeated failed log in attempts against one of the applications monitored by Fail2ban;
  • The free disk space on all partitions is less than a threashold established by the admin;
  • The average CPU load in the last 15 minutes exceeded 100% utilization of all the CPU cores;

System Health and Security Probe stores all the IP addresses banned by Fail2ban in a MariaDB database and if one IP has been banned more than once, it includes the ‘whois’ data for that IP in the email report. It also searches for the ‘abuse’ email address in the ‘whois’ data and writes a draft email that can be manually sent by the admin to the entity that owns the offending IP, to report the repeated attacks. The draft email includes the log lines containing the failed log in attepts for that IP. It can be configured to automatically send the abuse report email to the entity that owns the offending IPs that were banned more than once.

To install ‘System Health and Security Probe’ first create the /srv/scripts/shas-probe directory:

mkdir /srv/scripts/shas-probe

Then download the latest version of System Health and Security Probe from its official repository and upload all the files to the /srv/scripts/shas-probe folder. Set the right ownership for the new directory:

cd /srv/scripts
chown -R root:root shas-probe

Next, use phpMyAdmin to create a MariaDB database (called for example bannedipsdb), a user (called for example bannedipsuser) and a password, give the user all the priviledges over that database, except GRANT.

34.1. Configure System Health and Security Probe


Open the configuration file:

nano /srv/scripts/shas-probe/shsp-config.php

The first setting is the timespan in hours between two consecutive runs of the program. The default is 8 hours, but you can change it to any value you prefer. When using it for the first time, you can leave it to 8 hours, so make the
time_in_hours parameter look like this:

$time_in_hours = "8"

The next 3 parameters will mention the database name, database user and database user password. They shouldn’t contain double quotes:

$db_name = "bannedipsdb"
$db_user = "bannedipsuser"
$db_password = "strongpassword"

The next 3 parameters mention the sender’s email address, the recipient’s email address and the subject line. For multiple recipients, write all the email addresses separated by comma, like this: “admin@example.com,admin2@secondsite.info,admin3@thirdsite.org“. So, the report_from parameter specifies the email address which will appear in the ‘From’ field of all the email reports sent by System Health and Security Probe, while the report_to parameter specifies the email address or addresses of the admin(s) to which the reports will be sent:

$report_from = "admin@example.com"
$report_to = "admin@example.com"
$report_subject = "System Health and Security Report"

Next, set the log paths for the services monitored by Fail2ban. For WordPress, you can specify multiple log files, one for each website, separated by commas, like below:

$phpmyadmin_log = "/var/log/sites/mail.example.com/access.log";
$mailman_log = "/var/log/sites/mailman.example.com/access.log";
$dolibarr_log = "/var/log/sites/doli.example.com/access.log";
$phplist_log = "/var/log/sites/lists.example.com/access.log";
$roundcube_log = "/var/log/sites/mail.example.com/errors.log";
$matomo_log = "/var/log/sites/stats.example.com/access.log";
$looladmin_log =  "/var/log/sites/office.example.com/access.log";
$postfix_admin_log = "/var/log/sites/mail.example.com/access.log";
$roundpin_log = "/var/log/sites/roundpin.example.com/access.log";
$mybb_log = "/var/log/sites/forum.example.com/access.log";
$friendica_log = "/var/log/sites/friendica.example.com/access.log";
$redscarfsuite_panel_log = "/var/log/sites/panel.example.com/access.log";
$wordpress_log = "/var/log/sites/example.com/access.log, /var/log/sites/secondsite.net/access.log, /var/log/sites/thirdsite.info/access.log"

The following parameters specify standard log paths. If you haven’t changed them to some custom paths, leave them as they are:

$asterisk_log = "/var/log/asterisk/messages";
$bind_log = "/var/log/named/security.log";
$nextcloud_log = "/var/log/nextcloud/nextcloud.log";
$dovecot_log = "/var/log/mail.log";
$postfix_log = "/var/log/mail.log";
$postfix_sasl_log = "/var/log/mail.log";
$proftpd_log = "/var/log/proftpd/proftpd.log";
$sshd_log = "/var/log/auth.log";
$openvpn_log = "/var/log/openvpn/openvpn.log";

The next parameter specifies the threshold for the disk free space in kibibytes. If the free disk space becomes smaller than the threshold, it will be included in the email report as a system health problem. The default is 3145728 kibibytes (3 GiB):

$disk_threshold = "3145728";

Next, set the directory where the ClamAV antivirus will store the two periodic scan reports: ‘clamav_mail_report’ and ‘clamav_nextcloud_report’. The default is /srv/scripts/detections:

$clamav_report_dir = "/srv/scripts/detections"

The next parameter specifies the path of the backup directory where the current database will be automatically saved when the number of rows of the ‘bannedipstable’ table will exceed 100000000. After the current database is saved, the oldest 1000000 records of the ‘bannedipstable’ table will be trimmed. Set this directory name without a trailing slash. The default is /srv/backup:

$backup_directory = "/srv/backup"

Then specify if the script will send automatic abuse report emails to the Internet Service Providers (ISPs) of the offending IPs. Default is “no”. If you want ‘System Health and Security Proble’ to automatically send the abuse report emails to the ISPs of the offensing IPs, when the offending IPs have been banned more than once, each time the number of bans increases by 1, change “no” to “yes”:

$automatic_emails_to_isp = "no"

If the preceding $automatic_emails_to_isp parameter is set to “yes”, specify the list of log parameters (you can take any log parameter from the ones listed above), separated by commas, that you want to exclude from the automatic emails sent to the Internet Service Providers of the offending IPs. If you list a log parameter below, the automatic abuse report emails will NOT be sent to the ISPs of the offending IPs that attacked the corresponding jail. Here it’s recommended to list all the log parameters for which the log entries may contain usernames used for HTTP authentication or other sensitive data that you don’t want to be automatically sent to the ISPs in the abuse report emails. Thus, add or remove any log parameter to the list from below:

$excluded_jails = [$phpmyadmin_log, $wordpress_log, $roundcube_log, $dolibarr_log];

If the $automatic_emails_to_isp parameter from above is set to “yes”, specify the list of IPs, separated by commas, that you want to exclude from the list of IPs for which the abuse email will be automatically sent to the Internet Service Providers:

$excluded_ips = ""

If the $automatic_emails_to_isp parameter is set to “yes”, specify the sysadmin name that will appear in the signature of the emails sent to ISPs (Eg: $sysadmin_name = "John";):

$sysadmin_name = "";

If the $automatic_emails_to_isp parameter is set to “yes”, specify if you want the recipient of the email reports (the $report_to address from above) to receive copies of all the abuse reports sent to the ISPs of the offending IPs (if you want, change no to yes):

$abuse_reports_to_admin = "no";

Next, change permissions for the shsp-config.php file:

cd /srv/scripts/shas-probe
chmod 600 shsp-config.php

Add a cronjob to run System Health and Security Probe every 8 hours (or with a different frequency if you prefer):

crontab -e

Add the following two lines at the end of the file:

# Run System Health and Security Probe every 8 hours
5 */8 * * * php /srv/scripts/shas-probe/system-health-and-security-probe.php > /dev/null 2>&1

While the crontab file is open, check if you have the lines that tell ClamAV to scan two sensitive directories. They should have been added when installing Nextcloud, as explained in the Install Nextcloud chapter. The lines are:

# Scan the '/var/www/cloud.example.com/data' directory and the '/var/vmail' directory with ClamAV every three days
20 4 */3 * * cat /dev/null > /srv/scripts/detections/clamav_nextcloud_report && clamdscan --fdpass --quiet /var/www/cloud.example.com/data -l /srv/scripts/detections/clamav_nextcloud_report
10 5 */3 * * cat /dev/null > /srv/scripts/detections/clamav_mail_report && clamdscan --fdpass --quiet /var/vmail -l /srv/scripts/detections/clamav_mail_report

If you don’t have these lines, add them, replacing example.com with your own domain and save the crontab file. You can change the /srv/scripts/detections directory (where the periodic ClamAV scan reports will be stored) and the time and frequency of scanning, according to your needs, but keep the file names clamav_nextcloud_report and clamav_mail_report as they are. If you change the /srv/scripts/detections directory in the lines shown above, don’t forget to create the new directory if it doesn’t exist and specify it in the /srv/scripts/shas-probe/shsp-config.php file, in the $clamav_report_dir parameter.


After following the steps described above, the program will be ready to fulfil its mission. Please note that the email client (Eg.: Thunderbird) must accept HTML emails in order to properly display the email reports sent by System Health and Security Probe, which will look like this:

34.2. Upgrading System Health and Security Probe

To upgrade ‘System Health and Security Probe’ first make a backup copy of the old version:

cd /srv/scripts
tar cf shas-probe-2024-5-24.tar.xz shas-probe

Change 2024-5-24 with the actual date of the backup. Then make a backup copy of the database: use phpMyAdmin to export the bannedipsdb. Rename the exported sql file including the date in its name, then place the archive made earlier and the database backup in a safe location. Also, copy the /srv/scripts/shas-probe/shsp-config.php file to your computer to have it later when configuring the new version of the application.

Next, download the new version from the official repository, upload all the files to the /srv/scripts/shas-probe directory overwriting the old files with the new ones , then change ownership:

cd /srv/scripts
chown -R root:root shas-probe

Reconfigure the application, adding all the data from the old shsp-config.php file to the new shsp-config.php file, then set correct permissions for shsp-config.php:

cd /srv/scripts/shas-probe
chmod 600 shsp-config.php
You can send your questions and comments to: