System Health and Security Probe is a program that runs periodically and sends a report email to the administrator if it detects at least one of the following problems:
- One or more services running on the host server are in failed state;
- The periodic ClamAV scan has detected any viruses in the mail directories or in the Nextcloud upload directories, or the logs show recent virus detections in the incoming emails or in the files uploaded to Nextcloud;
- Any new IP address has been banned during the last run cycle interval due to repeated failed log in attempts against one of the applications monitored by Fail2ban;
- The free disk space on all partitions is less than a threashold established by the admin;
- The average CPU load in the last 15 minutes exceeded 100% utilization of all the CPU cores;
System Health and Security Probe stores all the IP addresses banned by Fail2ban in a MariaDB database and if one IP has been banned more than once, it includes the ‘whois’ data for that IP in the email report. It also searches for the ‘abuse’ email address in the ‘whois’ data and writes a draft email that can be manually sent by the admin to the entity that owns the offending IP, to report the repeated attacks. The draft email includes the log lines containing the failed log in attepts for that IP. It can be configured to automatically send the abuse report email to the entity that owns the offending IPs that were banned more than once.
To install ‘System Health and Security Probe’ first create the /srv/scripts/shas-probe directory:
mkdir /srv/scripts/shas-probe
Then download the latest version of System Health and Security Probe from its official repository () and upload all the files to the /srv/scripts/shas-probe folder. Set the right ownership for the new directory:
cd /srv/scripts
chown -R root:root shas-probe
Next, use phpMyAdmin to create a MariaDB database (called for example bannedipsdb), a user (called for example bannedipsuser) and a password, give the user all the priviledges over that database, except GRANT.
Change permissions for the config file:
cd /srv/scripts/shas-probe
chmod 600 config
Add a cronjob to run System Health and Security Probe every 8 hours (or with a different frequency if you prefer):
crontab -e
Add the following two lines at the end of the file:
# Run System Health and Security Probe every 8 hours
5 */8 * * * php /srv/scripts/shas-probe/system-health-and-security-probe.php > /dev/null 2>&1
Next, create the directory to store the periodic ClamAV scan reports. The default name and location for this directory is /srv/scripts/detections but you can change it and mention it as such in the config file.
cd /srv/scripts
mkdir detections
Then set up two cronjobs to get ClamAV to scan two sensitive directories. You can change the directory /srv/scripts/detections and the time and frequency of scanning, according to your needs, but keep the file names clamav_nextcloud_report and clamav_mail_report as they appear below. Replace example.com with your domain:
# Scan the ‘/var/www/cloud.example.com/data’ directory and the ‘/var/vmail’ directory with ClamAV every three days
20 4 */3 * * cat /dev/null > /srv/scripts/detections/clamav_nextcloud_report && clamdscan –fdpass –quiet /var/www/cloud.example.com/data -l /srv/scripts/detections/clamav_nextcloud_report
40 4 */3 * * cat /dev/null > /srv/scripts/detections/clamav_mail_report && clamdscan –fdpass –quiet /var/vmail -l /srv/scripts/detections/clamav_mail_report
34.1. Configure System Health and Security Probe
Open the configuration file:
nano /srv/scripts/shas-probe/config
The first setting is the timespan in hours between two consecutive runs of the program. The default is 8 hours, but you can change it to any value you want. When using it for the first time, you can leave it to 8 hours, so make the [timeinhours] parameter look like this:
[timeinhours] = “8”
The next 3 parameters will mention the database name, database user and database user password. They shouldn’t contain double quotes:
[bannedipsdb] = “bannedipsdb”
[bannedipsuser] = “bannedipsuser”
[bannedipspswd] = “strongpassword”
The next 3 parameters mention the sender’s email address, the recipient’s email address and the subject line. For multiple recipients, write all the email addresses separated by comma, like this: “admin@example.com,admin2@secondsite.info,admin3@thirdsite.org“. So, the [emailsender] parameter specifies the email address which will appear in the ‘From’ field of all the email reports sent by System Health and Security Probe, while the [to] parameter specifies the email address or addresses of the admin/admins to which the reports will be sent:
[emailsender] = “admin@example.com“
[to] = “admin@example.com“
[subject] = “System Health and Security Report”
Next, set the log paths for the services monitored by Fail2ban. For WordPress, you can specify multiple log files, one for each website; you must add the ‘wordpresslog’ parameter before the name of each WordPress log file, like below:
[asterisklog] = “/var/log/asterisk/messages”
[bindlog] = “/var/log/named/security.log”
[looladminlog] = “/var/log/sites/office.example.com/access.log”
[mailmanlog] = “/var/log/sites/mailman.example.com/access.log”
[dolibarrlog] = “/var/log/sites/doli.example.com/access.log”
[phplistlog] = “/var/log/sites/lists.example.com/access.log”
[phpmyadminlog] = “/var/log/sites/mail.example.com/access.log”
[roundcubelog] = “/var/log/sites/mail.example.com/errors.log”
[matomolog] = “/var/log/sites/stats.example.com/access.log”
[wordpresslog] = “/var/log/sites/example.com/access.log”
[wordpresslog] = “/var/log/sites/secondsite.net/access.log”
[wordpresslog] = “/var/log/sites/thirdsite.info/access.log”
The following parameters specify standard log paths. If you haven’t changed them to some custom paths, leave them as they are:
[nextcloudlog] = “/var/log/nextcloud/nextcloud.log”
[dovecotlog] = “/var/log/mail.log”
[postfixlog] = “/var/log/mail.log”
[postfix-sasllog] = “/var/log/mail.log”
[proftpdlog] = “/var/log/proftpd/proftpd.log”
[sshdlog] = “/var/log/auth.log”
The next parameter specifies the threshold for the disk free space in kibibytes. If the free disk space becomes smaller than the threshold, it will be included in the email report as a system health problem. The default is 3145728 kibibytes (3 GiB):
[diskthreshold] = “3145728”;
Next, set the directory where the ClamAV antivirus will store the two periodic scan reports: ‘clamav_mail_report’ and ‘clamav_nextcloud_report’. The default is /srv/scripts/detections:
[clamavreportdir] = “/srv/scripts/detections”
The next parameter specifies the path of the backup directory where the current database will be automatically saved when the number of rows of the ‘bannedipstable’ table will exceed 100000000. After the current database is saved, the oldest 1000000 records of the ‘bannedipstable’ table will be trimmed. Set this directory name without a trailing slash. The default is /srv/backup:
[backupdirectory] = “/srv/backup”
Then specify if the script will send automatic abuse report emails to the Internet Service Providers (ISPs) of the offending IPs. Default is “no”. If you want ‘System Health and Security Proble’ to automatically send the abuse report emails to the ISPs of the offensing IPs, when the offending IPs have been banned more than once, each time the number of bans increases by 1, change “no” to “yes”:
[automaticemailstoisp] = “no”
If the preceding [automaticemailstoisp] parameter is set to “yes”, specify the list of jail log names (you can take any jail log name from the ones listed above), separated by commas, that you want to exclude from the automatic emails sent to the Internet Service Providers of the offending IPs. If you list a jail log name below, no automatic abuse report email will be sent to the ISPs of the offending IPs that attacked that jail. Here it’s recommended to list all the jail log names for which the log entries may contain usernames used for HTTP authentication or other sensitive data that you don’t want to be automatically sent to the ISPs, in the abuse report emails. Thus, add or remove any jail log name to the list from below:
[excludedjails] = “phpmyadminlog,roundcubelog,wordpresslog”
If the [automaticemailstoisp] parameter from above is set to “yes”, specify the list of IPs, separated by commas, that you want to exclude from the list of IPs for which the abuse email will be automatically sent to the Internet Service Providers:
[excludedips] = “”
After you save the config file the program will be ready to fulfil its mission. Please note that the email client (Eg.: Thunderbird) should accept HTML emails in order to properly display the email reports sent by System Health and Security Probe.
34.2. Upgrading System Health and Security Probe
To upgrade ‘System Health and Security Probe’ first make a backup copy of the old version:
cd /srv/scripts
tar cf shas-probe-2020-5-24.tar.xz shas-probe
Change 2020-5-24 with the actual date of the backup. Then make a backup copy of the database: use phpMyAdmin to export the bannedipsdb.Rename the exported sql file including the date in its name, then place the archive made earlier and the database backup in a safe location. Also, copy the /srv/scripts/shas-probe/config file to your computer to have it later when configuring the new version of the application.
Next, download the new version from the official repository, upload all the files to the /srv/scripts/shas-probe directory overwriting the old files with the new ones , then change ownership:
cd /srv/scripts
chown -R root:root shas-probe
Reconfigure the application, adding all the data from the old config file to the new config file, then change permissions:
cd /srv/scripts/shas-probe
chmod 600 config
