• Twitter
  • LinkedIn

Blog

Downtime: A Cautionary Tale of Technology

January 27, 2015   By: Richard Raudon

How protected is your business against technology failures? What would happen to your business today if you had an outage of your accounting, or payrolling, or recruitment systems?

Many legacy systems are vulnerable to technology failure. Cloud based solutions, by comparison, offer a very viable alternative – cloud providers invest far more in the resilience and ‘hardness’ of their infrastructure, meaning their solutions and services are far less likely to experience outages.

We have a cautionary tale about what happens when legacy systems stop working.

The background

The Australian recruitment market is large, but compact. There’s a lot of interconnection. Many of our current customers were once leads. We know how important it is to nurture leads, keep in contact and keep them engaged.

As such, we invest in communication with our leads and contacts. We work hard to keep connected with our clients, both past and present, and to keep engaged with our potential clients.

In late 2014 we undertook an exercise to update all our lead details, ensure everyone was still keen to hear from us, and keep them up to date.

It took about a week to set everything up, from email templates to Salesforce reports to targeted content. To kick off the digital part of the campaign, we used ExactTarget to send out a brief, targeted email to our clients.

After the campaign had kicked off,  I got a call from Simon Bishop, from whom the email had been sent. ‘Don’t want to alarm you, but I’m getting responses that the website’s down. Can you look into it?’

I was somewhat surprised – we’d never had a problem with web connectivity before – and hurried to check it.

Noone home

500: Internal server error. Cannot connect to database. Please contact the administrator (admin@2cloudnine.com).

Well, that’s definitely an outage. Admin panel? Same error. Web hosting panel? Unavailable. Yet our provider’s uptime indicator showed everything was live.

I call through to our provider. After fifty minutes on hold, I am escalated to level 2. Another forty minutes later, I’m advised that the site is indeed down and will be fixed in two to six hours.

In the meantime, we’re watching the email activity reports and seeing multiple clicks to what we know is a dead site. But what do we do? Re-email apologising for the outage, with no site to redirect them to? Set up a new hosting account and migrate the content across? None of the above – with an anticipated two to four hours downtime, we made the call to wait till the site is restored, then re-send the initial email.

What we didn’t know is that our web provider was in the process of migrating customers from its legacy hosted systems to the cloud. From the feedback spotted on the company’s social profile pages and the Whirlpool.net forums, the hosting for a number of clients just plain stopped working – ours included.

The waiting game

Two hours elapsed, with no change. Then four. Then six, eight, ten, twelve. Multiple phone calls went unanswered at 30 minutes apiece. The following morning, with twenty hours elapsed, I awoke to more messages from the team. Site’s still not up.

After another marathon hold session – seventy minutes, this time – and I get through to the support desk, to be told the problem had not progressed but is being worked on.

In an attempt to build some basic presence, I used week-old backups of the site, restored it to our clunky, slow development server, and pointed the domain name to it. It worked – just.

A full 26 hours after the initial outage, the site reappeared. Not the current version – an archived version, older than our own backups. Another eighty minutes on the phone to ask why and how and what could be done to prevent a repeat, with little in the way of comforting advice.

 

The outcome

We took the hint, and quietly set up a cloud instance of our website, directly through Amazon Web Services. Which, incidentally, backs itself up and restores the backup automatically if it notices that the instance has gone down.

We got away relatively lightly – a bit of egg on the corporate face, as it were, for a non-performing website. There was no operational impact, and thanks to some nifty features in our digital marketing product, were able to follow up individually with people who clicked – and missed – the site.

But we still lost 26 hours worth of eyeballs across our website and further opportunities to engage with our leads.

Discussion points

Our provider was a reputable, Australian-based technology firm, with a talented local support team, that came highly recommended to us. Yet at crunch time they were unable to deliver on their SLAs, which has had an impact upon our business.

That is the risk of working with organisations using dated technology. Cloud based solutions offer far more in the way of stability and resilience than legacy systems.

Consider the impact that a technology outage could have on your business – what would be the cost to productivity? To reputation, brand and revenue?

Can you afford to accept this risk?


Richard RaudonRichard Raudon, Marketing Manager at 2cloudnine. Connect with Richard Raudon on LinkedIn

 

The Acumen Series | Excelling in Modern Business Development
We’re hiring! 2cloudnine searching for a Solutions Architect