Since this guide is complete, it includes the maintenance steps that should be followed in order to keep the server in good condition and up-to-date.
1. Once a week, typically every Friday, download the backups from Nextcloud to two other places: one can be your laptop/desktop, and the other can be an external hard drive. A safe backup exists in three copies: one typically on the machine whose data is backed up, in a special directory with restrictive permissions, so as to have easy access to it when necessary, a second copy on your personal laptop/desktop to which you have exclusive access, and a third copy on an external hard drive only you can access. The golden rule is that the three copies of the backups should be stored in places situated physically far apart from each other, but in such a way that they would be easily accessible to only one person: you. Of course, your laptop/desktop and the external hard drive may be close to each other, but the distance between your computer and the remote server should be substantial, so that an accident affecting one would not affect the other. It’s not advisable to store your backups in more than three places because this increases the odds that they will fall into the hands of the wrong people. And you have to remember that the backups include all your important data, including sensitive information such as certain passwords. So, a backup in three different places means safety, while more than three backups actually decrease safety progressively.
Let’s suppose you rented three VPSs in the same data center of a hosting company, then you stored a copy of your backups on each of those VPSs and then conclude your data is safe. That’s wrong. Imagine what will happen if that entire data center is destroyed due to human error or natural disasters such as earthquakes, hurricanes, floods, landslides, volcanic eruptions, wildfire, etc.: all your data on the live websites and all your backups will be lost in minutes, along with months or even years of work. This is why backups need to be stored in multiple places physically located far apart from each other.
Estimated time if no errors occur: about 10 minutes per week (or 8 hours per year).
2. Once every four weeks, log in to your server via SSH, ensure that there are no services in failed state with the systemctl --failed
command, browse through the following logs:
/var/log/syslog
/var/log/mail.log
/var/log/mail.err
/var/log/dovecot.log
/var/log/ufw.log
/var/log/letsencrypt/letsencrypt.log
/var/log/asterisk/messages
/var/log/fail2ban.log
/var/log/nextcloud/nextcloud.log
/var/log/named/security.log
/var/log/nginx/example.com.error.log
and other similar logs, to see if there are any errors, warnings, malfunctions, etc., then perform all the necessary upgrades with the apt-get update && apt-get dist-upgrade
command.
Then, log in to WordPress and apply all the themes and plugins updates (if any) as described in the Upgrading WordPress themes and plugins chapter, then update WordPress itself, if a major update is available, as described in the Upgrading WordPress chapter. Please note that we recommend applying all the updates first on the dev website and then on the live website. Also log in to Nextcloud and update all the apps that have updates available.
Next, check on their respective official websites if there are new versions for the following applications: Postfix Admin, Roundcube, phpList, phpMyAdmin, Dolibarr, Roundpin, MyBB, Friendica, Matomo, System Health and Security Probe, RED SCARF Suite Panel, and if necessary, perform the upgrade following the upgrade steps described in the chapters dedicated to these applications. (As mentioned earlier, if no critical security vulnerabilities are discovered, Asterisk, Collabora Online Development Edition (CODE) and Nextcloud should be upgraded only once every 2 years, after you upgrade Debian.)
Please note that you should perform the upgrade for a specific application only after you make sure that its newest version works well in the context of the entire software suite described in this guide. To find out which is the newest version of an application that has been tested and confirmed to work well within the suite, you can check if it has been included on the list available on this page.
Please keep in mind that the upgrading step is the point where much too often beautiful dreams turn into nightmares. Applications that worked flawlessly for years can become defective or even crash because of an upgrade. This usually happens because applications are so complex that it becomes very difficult for their developers to test them in all possible scenarios. Even when new versions are tagged as ‘stable’ they can still contain undetected bugs. That is why you have to know in advance if a new version of an application is really ready to be installed.
If you want to be informed when new versions of applications have been released and are safe to install, when critical security vulnerabilities have been discovered in any of the applications that make up RED SCARF Suite and when the text of this guide gets updated, you can subscribe to our ‘Updates Newsletter’.
Estimated time if no errors occur: about 1 hour per four weeks (or 13 hours per year).
3. Once every two years, when you find out from this page that a new Debian stable version is released, you will have to upgrade Debian to the newest version, upgrade certain applications and take additional maintenance steps, by carefully following these steps:
- inform all your users, at least one week in advance, by email or in a banner displayed on websites/applications, about the precise date and hour when the upgrade will be scheduled and that the provided services will be unavailable at that specific moment.
- ensure that you have a second way of accessing your server besides SSH, in case the worst thing happens during the upgrade: you remain locked out of your server, which becomes unaccessible via SSH due to some network configuration that gets changed, etc. Some hosting providers offer a web console that allows accessing a server without using SSH, while others offer a rescue mode boot up that you can use to log in to a different functional server from which you can mount the hard drive of your disfunctional server, edit any files as needed, and then you can restart your server to apply the new changes.
- next, upgrade Debian by carefully following the steps presented in the Upgrading Debian to the newest version chapter of this guide.
- check if everything is working well, if there are any services in failed state, etc.
- download the latest version of the ‘Memcached Object Cache’ WordPress plugin from
https://wordpress.org/plugins/memcached/
. Uncompress the archive, create a random string to use as salt by runningopenssl rand -base64 48
, add the salt at the beginning of theobject-cache.php
file, like this:
if ( ! defined( 'WP_CACHE_KEY_SALT' ) ) {
define( 'WP_CACHE_KEY_SALT', 'vkCi6QNmBT4drFMXpCIj57iEGte54CGO9yathz8fkdAsfP0k2gOU6lGdbLGHLqSh' );
}
then replace the old /var/www/example.com/wp-content/object-cache.php
file with the new object-cache.php
file that you have just edited, for each website for which you use Memcached for object caching.
- Let’s Encrypt stores the old SSL certificate files in the directories of the form
/etc/letsencrypt/archive/www.example.com
. Once every two years you can delete the old certificate files (having the form:cert123.pem
,chain123.pem
,fullchain123.pem
,privkey123.pem
), because they get more numerous every three months and once they have expired, they are not needed. Pay attention not to delete the most recent files which are needed. The most recent files have the highest number at the end of their name, before thepem
extension.
- build and install Asterisk again, as described in the Build and Install Asterisk chapter. This is needed because Asterisk depends on software packages that in general change from one Debian version to the another.
- upgrade Collabora Online Development Edition (CODE), as described in the Upgrading Collabora Online chapter.
- upgrade Nextcloud, as described in the Upgrading Nextcloud chapter.
Estimated time if no errors occur: about 4 hours per two years (or 2 hours per year).
Total estimated time for periodic server maintenance if no incidents happen: about 23 hours per year, which gives an average time of about 2 hours per month for each server on which the standard RED SCARF Suite is installed.