Prevention beats cure: why maintenance matters so much

You probably back up your data, but how often do you check that your backups work? Are you aware when your server is about to run out of space? What would you do if your system went down and wouldn’t come back up? We’ve thought hard about these questions and we have some answers.

In the early days of our business, the phone used to ring every morning. ‘Help, our system’s gone down and we can’t do any work!’ We would drop everything to sort it out. We felt like an emergency recovery service, which was great, but after a while we knew something wasn’t right. We weren’t preventing things that could be prevented, and that was poor service.

Let’s face it, your server is a critical part of your business infrastructure. You can’t afford for it to fail on you, even if you have it up and running by the end of the day. That’s a day of poor customer service, missed orders and delays. And if you lose the last six months’ data, that’s really serious.

The fact is, just saving your data into your system doesn’t mean it’s actually safe. So we created our own gold standard preventative maintenance service for servers. It’s an unsung operation, but the actions we take are wide-ranging, complex and important.

Regular backups

First, we take regular backups, and that means daily. Every night, the servers we look after send us a copy of their entire system. These servers are a mix of those we run as part of our hosting service and many that belong to and are run by our clients. This data is automatically sent to us encrypted and then securely stored. Each night we receive and store more than 700GB of data from dozens of businesses. This stored data is also backed up remotely just in case something happened to our servers.

Checking the backup

Many people have discovered to their horror that when they try to open a backup (which they’ve dutifully and regularly saved) it doesn’t work. The data you desperately need can’t be opened. We make sure this will never happen by opening every backup and checking it works. We used to do this twice weekly and now we do it even more often.

People also sometimes find that they can open their backup, only to find that it’s actually six months old. This isn’t uncommon. So we also make sure each backup is a new copy, which we do by checking that the data is different from the previous backup. This indicates it’s a genuine copy of the new state of the server. If the data hasn’t changed, either there’s a problem or for some reason the system hasn’t been used – whichever it is, we follow up to find out.

Server checks

We also carry out a series of checks to ensure the server is in good health. We make sure that there’s enough disk space left on the server, the server has been periodically restarted when it needs to be, the RAM is working, and carry out other hardware checks. We also ensure certain key pieces of data are up to date – for example, for a US client we update tax rates every three months – along with SSL certificates.

A precisely managed programme

These processes take place at different intervals – some daily, others weekly, monthly, quarterly or annually. In fact there are currently 397 recurring tasks each month.

This complex web of checks is scheduled by a FileMaker system that we have built for the purpose. It tells us what needs doing, when and by whom – and it records the outcomes, along with evidence the task has been done, such as a screen shot. In August 2024 we completed 2,067 individual maintenance tasks.

Today we get very few of those panic phone calls because we prevent the problems at source. But there are other benefits too. If a client’s server fails, we can use the previous day’s backup to get their system back online, often using a hosted server while their on-site server is being repaired.

We also use backups for testing and fault-finding. If someone reports a fault in their system, the first thing we do is try to replicate it. But rather than doing so in their live system, we use the latest backup as a test system – we already have a copy of it on our network, so there's no download time.

Another step forward

Recently we’ve added another set of improvements. Our preventative maintenance programme was easy to implement with a few clients, but as we’ve grown it’s become a bigger and bigger job. One of our developers – taking to heart our ethos of solving problems when you see them – took it upon himself to develop a set of tools that automates the process of taking backups, checking they are valid backups and verifying that they work. The system then issues a report on its findings and highlights any issues, which we can see as soon as we start work in the morning. There’s no delay between a problem occurring and the effort to solve it.

The automation is a clever solution that called for creative development work, and it means we can now run these checks every working day, adding an extra layer of security – but with far less manual work. It is a good example of the problem-solving solutions we create for our clients – it’s just that this time we developed it for ourselves. It has already saved around 130 person-hours a month, which equates to virtually a whole member of staff. The saving allows us to focus on creative work rather than repetitive manual tasks, and make it easier for us to scale and grow.

This entire server maintenance service is the most advanced programme of its kind that we know of (and we’re still refining it at our monthly maintenance improvement meetings). We’re confident that no other process matches ours for completeness and dependability. When it comes to protecting your data, this is the gold standard.