Hands up if, as a change manager, you’ve seen some truly horrendous change requests?
Changes so mangled and broken that their only conceivable purpose could be to serve as a dreadful warning to other change requests to straighten up and get a job.
We are occasionally labelled as the ‘parking wardens of the ITSM world’. That’s not to say we’ll invent improbable and eye-watering fines, but we are on the lookout for likely offenders and we’ll be consciously (and sometimes subconsciously) assessing your change requests against a ‘bingo card’ of suspicious behaviour when giving each request an initial quality check.
Good quality changes present the right information to the right people to make the right decision – so what are common reasons for rejection at that first quality gate?
Dear Requestor – your change has been rejected because:
- ‘N/A’. – Change forms are pretty generic, we get that – but the minimum we’re looking for here is a sensible reason why it isn’t applicable.
- Risk/Impact = ‘None’ – If you’re touching production, I’d argue that the risk is never none. I’ll accept ‘negligible risk to production based on rehearsal test results’ or ‘no material impact to key business services; isolated on a separate vLAN’ but I’ve seen far too many ‘harmless’ changes killing production.
- ‘TBC’. – Ok, we’re not totally unreasonable, it takes time to get some information and we know you only had 20 minutes after identifying the need for change to get it in before the weekly CAB cutoff, but when will it be confirmed? What information are you waiting for? Will you have it in time for CAB?
- Leaving blanks. If you’ve submitted it with key information missing (why, what, where, when, who & how) then we’d find it difficult to ask CAB to make a good decision based on missing information. Cover the basics well enough, and you might get occasional slight offences overlooked (especially if you arrive at CAB with delicious pastries).
- Suspiciously short answers – ‘Rollback’ is not a remediation plan. ‘Rollback to last snapshot taken at start of deployment. Takes 10 mins, will cause an outage and need 30 mins further checks afterwards by the DBAs. Rollback has been used in the past with no issues’ is a much better starting point.
- Suspiciously long answers – Just like the overdue motorist who starts winding up a long, complex and improbable excuse, if you’ve copied and pasted 37 pages of vendor release notes into a text field we’re going to examine the rest of the change even more carefully. By all means attach supporting documentation and give a summary. This one leads me directly to:
- Change descriptions comprising only code – Look, I get that you’re smarter than me when it comes to development, query plans, subnetting, or many other fields of specialty. I’ve even had a change request comprising only an algorithm for a Kalman filter – which even experienced statisticians regard as voodoo. I’m looking for reasons to trust that you know what you’re doing when you raise a request. If you can wrap your code snippet in plain english to describe the problem it fixes and (at a high level) how, then we’re good. I’ll also understand more about your change which means I can help you make a case for it. We’d rather help than hinder.
- “Step 1 – Do the change. The End.” If you look back at this in 6 months time, because a sneaky recurring problem started at about the same time and has been driving you insane trying to figure out what caused it, will you know what it was you did? And how do we trust that you really know what you’re doing? Even simple changes have more to them than ‘just do it’. (You’ll thank me for this at midnight on a friday in about a year from now.)
Here’s an example of an implementation plan for a really simple patch:
- Obtain patch from vendor site at www.vendor.site/patch-id=12345
- Extract binaries, verify release notes and checksum
- Check service & server monitoring for unexpected issues which may impact release. Escalate to Duty Ops Mgr if in doubt.
- Stop application service
- Deploy patch following (attached) vendor instructions with the following deviations [xxxxxxx]
- Check logfiles for [xxxxxxxx] errors
- Restart application service
- Check service & server monitoring after change
- Close change record and hand back to ops.
Why is missing a few bits of information such a drama?
Because change management is a decision game. And the only way to consistently win decision games is to make decisions based on the best information possible. If you don’t give us this information, we’ll make bad decisions which ultimately expose the business to unnecessary risk due to operational instability or sacrifice our responsiveness to changing business demands. Or to be brutally direct: garbage in, garbage out.
It’s unlikely that poor change requests are the result of malicious individuals (unless you’re really unlucky). It’s also overly-simplistic to call it laziness:
“Ordinary laziness was merely the absence of effort. Victor had passed through there a long time ago, had gone straight through commonplace idleness and out on the far side. He put more effort into avoiding work than most people put into hard labor.”
~ Terry Pratchett, Moving Pictures
I’ve witnessed requestors put more effort into arguing why they shouldn’t have to raise a change request than they have into following the process in the first place. Luckily, these events are the exception rather than the rule, but as change manager, ask yourself (or others for an objective view) these questions:
- Is the change process logical, efficient, intuitive and easy to understand? How about the form/tool they’re logged in?
- Are there any unnecessary bottlenecks that can be engineered out?
- Is your approach to managing change proportionate? Do you have lighter processes for simpler and less risky changes? Not just Standard (catalogue) changes, but minor technical one-off changes that may suffice with a peer review and change manager approval* and don’t need the full process?
- Do people know what a good change request looks like? Do you have ‘gold standard’ example change requests in the back of your policy/process document?
- In fact, do they know where to find the policy/process documents? (hint – put a link to them in your email signature)
(*for anyone aghast that the change manager can approve a change ex-CAB, check out the ITIL(R) (2011) Service Transition core publication in section 126.96.36.199 which in figure 4.5 shows an example of a graduated approval structure. Not all Normal changes necessarily have to go to CAB if that’s what you agree in your policy, based on risk & impact.)
If your processes, tools, forms and model changes are shining beacons of efficiency, clear simplicity and proportionate governance, then you likely have a training or cultural issue. Cultural issues are too complex for this article to deal with, but if you’ve studied frameworks such as ITIL, you’ll have an idea of how to sell the benefits of industry good practice to the people in charge, and you also have the option to create your own culture within Service Transition.
Training is an easier topic. Work out your most important message, stick to it and keep repeating it. Training problems will keep re-appearing as new staff arrive and as people forget, so keep your training materials handy and up to date. New staff induction training is one area to consider – you’ll get them whilst they’re still excited about their new job and keen to please. If this is impractical, then mandatory change training can be given before new users are allowed to raise change requests in the system. Weekly email tips/reminders is something else I’ve found to be useful in some situations.
If they still don’t get it
I’m often asked what to do about repeat offenders. An important message is that you are not here to knock down their changes or waste their time, you want to show them how they can create changes which can be processed quickly and efficiently.
I’ve seen HR policies and public shaming used to identify & punish people not following process. But apart from gross negligence, threatening to tell HR can have unpredictable results, and public shaming simply creates an unhealthy culture.
My graduated approach now is:
First Offence / Requests for help
Sit down with the requestor (geography permitting) and explain what needs to be improved and why. You can even help them (re)write it. It’s time well spent to show them the professional respect for their time that you’d want in return for the change process. If you can do this even before they submit their first change, so much the better. Prevention is, after all, better than cure.
A shot across the bows
Return it to the requestor with a short description of the gaps and a link to the process, policy and ‘gold standard’ example change. Offer to help if they’re struggling to articulate part of their change request; sometimes they might just need introducing to the relevant subject matter expert or someone who’s recently delivered a similar change.
Repeat this step at your discretion.
Return it, but copying the requestor’s line manager as final warning informing them that the next step will lead to:
Remove the requestor’s ability to raise changes in the system via access control. It can only be reinstated by someone senior enough to cause the requestor discomfort in having to ask them.
I’ve rarely had to apply sanctions. But if handled correctly and objectively, it’s a proportionate response which is within the power of most Change Managers as a last resort.
The parking warden analogy I opened with tells only half the story. It bears repeating that this isn’t about stopping requestors or making their lives difficult, it’s ultimately about protecting the business and responding to their needs. To do that we need to be able to make good decisions based on accurate and complete change request information as efficiently as possible.
Perhaps a better metaphor would be that of an Air Traffic Controller. We want you to land safely, but if you can’t evidence that know how, or if you list your destination as ‘Not Applicable’, then we’re not even going to let you start the engines, let alone get off the ground.