In March 2014, US Retailer, Target revealed that its security software had detected its now infamous data breach five
months earlier, and that at least eight IT employees had seen the threat alert but decided not to act on it. Some commentators jumped on the firm for its apparent incompetence, but security experts say its reaction was pretty normal.
So how and why do data breaches, equipment failures and disasters go undetected by humans when the monitoring systems are doing their jobs? The constant stream of alerts can cause engineers to check out, a syndrome that some refer to as ‘alert fatigue’.
Reacting to this influx of alerts uses your engineers’ time and resources, costs money, and can prevent your IT department from playing a more strategic role at your company. This article will explore four actions that you can perform now to address alert fatigue.
Here are the four recommended actions.
Action One: Plan
You could think of a notification model in four levels of maturity, listed here from least to most mature:
- Level 1 – reactive
- Level 2 – tactical
- Level 3 – integrated
- Level 4 – strategic
IT is complicated enough. Your IT tech people and engineers receive a stream of notifications that range from innocuous (someone has accessed an asset or logged into a system) to important but only to certain people (a project has achieved a milestone) to urgent (a server is down or security has been breached). Responding – or even reacting – every time a notification comes up can be time-consuming and irritating.
Do the work on the front end: Plan for alerts, escalations and automated processes for different scenarios to make sure your intelligent communications work well. The system must have every stakeholder’s contact information, device preference, schedule and commitment to be available. You must build this in advance of an emergency.
Action Two: Automate
Suppose your business experiences a power outage. A full-scale emergency will require a series of manual instructions and emails to the IT team, engineering and everyone whose business and safety may be affected. However, you can still automate some important features, alerting first responders, letting purchasing know you need new servers, and even cutting off power to the server room.
What about more limited incidents, such as an employee laptop failure? Once the incident is recorded, the engineering tech replacing laptops receives an alert, a step that can be automated, and the employee can receive an automated notification that a fix is in progress. What if the employee reports the issue after hours? Do you alert the tech on a mobile device, or can it wait until morning? If you plan your processes well, you can automate every step based on the urgency of the incident.
Time is critical, especially if you are servicing employees in global offices, as some employees are losing valuable work time. That could mean sales opportunities missed or incomplete timesheets. Based on urgency, location, time, each person’s preferred device and work schedule, you can automate whether to alert engineers right away or wait until the morning. Depending on the rules in place, the message can be sent two ways: automatically triggered by the event, or at the push of a button, usually by the IT lead.
Action Three: Be Proactive
Another important function for efficiency is the enablement of easy status updates. IT techs frequently experience disruptions from answering queries on the status of an open ticket. Whilst it’s understandable that customers want to know the status of outstanding events, IT techs would rather be resolving issues than answering enquiries. Status updates send automatic messages to clients with expected time to resolution.
Proactive communications don’t have to be just for incidents. They can let employees know of impending software updates, let customers know of enhancements, or let an employee know a new laptop has been ordered and is on its way. The proactive alerts can ease the minds of the recipients, whilst freeing IT leaders from such enquiries.
Action Four: Target
A good way to enable your engineers to avoid alert fatigue is through targeted alerts, as alerts go to the subset of employees who need to know either to take action or to simply be in the know. You should also target alerts by preferred device, so IT techs receive notifications where they’ll see them and respond. A good way of doing this is with subscriptions, enabling stakeholders to subscribe to relevant alerts and unsubscribe from others. When you combine automation, targeted alerts and subscriptions, you create more efficient alerting processes to help support IT Service Managers and IT departments.
With these recommended actions you should be able to drastically reduce the number of alerts received and help to restore some energy into your alerting.