One of the things that I got asked about most as a consultant was about what the difference was between Change and Release Management. It should be simple right? We’ve had 3.5 versions of ITIL, the itSMF, even special interest groups dedicated to Service Transition yet there is near universal confusion at the sharp end about WHAT the difference is between Change and Release Management so let’s sort this out once and for all!
For my money; Change Management are guardians – they protect the live environment.
The primary objective of Change Management is to enable beneficial Changes to be made, with minimum disruption to IT Services.
Release Management are like air traffic controllers; they package together bundles of Change into a single Release to reduce periods of downtime and inconvenience to the rest of the business.
The primary objective of Release Management is to ensure that the integrity of the live environment is protected and that the correct components are released.
Bringing out the big guns
Let’s get back to basics and talk ITIL for a second. Both Change and Release Management sit in the Service Transition stage of the ITIL lifecycle so are part of the value stream that delivers effective business change;
Change; The addition, modification or removal of anything that could have an impact on IT services.
Release; Collection of hardware, software, documentation, processes or other components required to implement one or more approved changes to IT Services. The contents of each release are managed, tested and deployed as a single entity.
In other words, Change is about installing, modifying or retiring things safely without setting anything on fire. Release Management is a holistic process that bundles together multiple Changes into a single deployment. So now that we’ve got that sorted; let’s talk about how to make sure Change and Release Management play nicely together.
Have a way of highlighting Releases in your Change Management tool
This ensures that Releases should up in the Change Schedule (CS) and that everyone is aware of any major deployments. This will make sure there are no conflicts or scheduling clashes. Take it from someone who knows there is nothing more uncomfortable than being on site in central London explaining to several different technical teams why a site power down and a major code deployment at the same site can’t go ahead at the same time. #awkward
Separate the roles of Change Manager and Release Manager
Change Management is a governance process, the role of the Change Manager is to review, authorise and schedule the Change. Release Management is an installation process. It works with the support of Change Management to builds, tests and deploy new or updated services into the live environment. Both are equally important so you need subject matter experts for both.
Agree the level of documentation required
A Change is a single record containing:
Approval details & audit trail
Release documentation is much more involved and as a starter for ten will contain:
Back out plan
Ensure the Release Manager is present at CAB
If your Release Manager isn’t attending CAB invite them immediately! It’s really important that the Release Manager is there to explain the Release content and any dependencies, communicate business approval and advise the Service Desk and Problem Management of any defects; working with them to ensure any known errors with any workaround details are raised where appropriate.
By having Change and Release Management working closely together, your effectiveness rates should improve and unforeseen incidents, problems and defects should be reduced. How do you manage Change and Release Management? Let us know in the comments!
*Yes; that’s an Aer Lingus plane; they are the best airline in the world because when the flight attendants clock that you’re a single mum with three over excited kids that are about to go feral at any moment and stress levels that are about to hit DEFCON1 they give you free vodka and cokes throughout the flight.
As a former Change Manager I can honestly say that the Change Advisory Board (or CAB) is one of the most important and useful meetings a service orientated organisation can have. It sets out a view of what’s happening to key services over the next week, reviews previous Change activity and looks at CSI so what’s not to like? CAB meetings are all about the people attending them and handled badly your CAB meeting will have all the power of a chocolate teapot so here are our top tips for running them effectively.
Step 1: TCB Power!
A colleague of mine once told me that TCB or tea, coffee and biscuits was one of the most important acronyms in IT. When I worked for a large investment bank in London, one of my first tasks was to roll out a sensible Change Management process across one of our service families. Trying to persuade grumpy techies who saw Change Management as red tape rather than an important part of service delivery was not going well until I brought out the big guns; Krispy Kreme doughnuts and chocolate biscuits. In all seriousness, a CAB meeting is where you want people to feel comfortable representing Changes or asking questions so anything that makes your meeting easier, nicer or makes people feel more relaxed can only be a good thing.
Step 2: Get organised
Make sure that your CAB has a terms of reference document so that everyone knows what they’re doing and why. The Change manager should send out the CAB agenda, including the Changes to be discussed, the Change Schedule (CS) and any Changes that caused Incidents well in advance of the CAB meeting. Service Delivery teams and Project Managers need time to read and consider the Changes as well as identify any potential issues or questions.
Step 3: Look for the big hitters
One of the biggest mistakes people make is insisting all Changes should go to CAB. Not a good idea unless you want your CAB meeting to be overrun with server reboots or patching requests. Use automation where possible so that the CAB meeting can focus on the major, high category Changes that need to be sanity checked and talked through.
Step 4: Play nicely with your attendees
Some members of the CAB will be needed for their opinion on every Change; for example the Service Desk, Network Services and Server support and will make up the core CAB attendee list. Other attendees such a Project Managers, Service Delivery Managers and external suppliers might only be needed to discuss a couple of Changes on the list. If this is the case then be kind. Move those Changes to the beginning of the CAB so that these temporary or “flex” CAB attendees can discuss the relevant Changes and then leave.
Step 5: Ask the horrible questions
You know the ones, what everyone in the room is thinking but no one wants to actually ask. Some examples could include:
“What’s the remediation plan? Do we fix on fail or roll back?”
“What happens if rolling back doesn’t fix the issue?”
“Is the person doing the Change empowered to make that decision or do we need to arrange for extra support to be on call?”
Or even; “is this really a good idea?”
It’s better coming from the Change Manager than from an angry customer or senior manager following a failed Change right? Make sure the Service Desk feel comfortable asking questions as well; they’ll be the ones at the sharp end of customer complaints if anything goes wrong so make sure they’re happy with the Change content and plan.
Step 6: Keep it pacey
There is nothing worse than a two hour CAB meeting. I guarantee you; if you are regularly putting your CAB attendees through marathon meetings then people will run short of both patience and good will. There’s also a very real chance that someone may fall asleep. Keep things moving. If someone has launched into a long winded, uber waffley technical explanation and you get the sense that it’s adding no value as well as making everyone in the room lose the will to live then break in with a question so that you can get things back on track. Do it nicely though obvs.
Step 7: CSI
Don’t forget to review your list of previously implemented Changes. If something’s gone well then brilliant! Let’s template it or add it to any Change models to share the love. If something hasn’t been successful or worse, has broken something generating a load of Incidents then look at what happened, figure out the root cause and look at ways of preventing recurrence. If your Problem Manager isn’t attending CAB then invite them – they are the subject matter experts in this area.
What do you think? What are your top tips for effective CAB meetings? Tell me in the comments!
Following on from my previous post about surviving implementation, here’s some advise on running your Change Management process aka The Day Job.
In Change Management your key areas to focus on are:
Recording and processing the Change
Change Advisory Board (CAB)
Build and Test
Raising the Change
So first up; recording the Change. Ensure your Change Management policy covers who can request a Change i.e. can anyone raise a Change? Just IT? What about the business? Each organisation will be different but one thing to keep in mind is making sure your policy gives clear guidelines on the difference between a Change and a Service Request i.e. with Request Fulfilment ends and where Change Management begins.
Create a Change form so that Changes can be raised in a standard way. It’s really important to have consistent information in your Change requests so when reviewing and approving Changes, you have all the facts needed to make the right decision for your customers. If you have a sparkly, all powerful toolset to do it for you then happy days. If not, and there are lots of organisations just getting started with Change Management, then it’s time to get creative. In the past, I’ve set up Change request forms using Word or Excel (tweet me if you would like to see some examples).
Things to consider for your form:
Will the CMDB (if you have one) need to be updated afterwards
Pre implementation testing
Post implementation verification
Back out plan
Impact to other environments
Will the change be replicated to your DR environment?
You need to have a lifecycle approach for raising Changes for example; the process for a standard Change will be very different to an emergency, sorting something out in the middle of the night type Change. Ensure this ties back in to your policy with criteria and examples so there’s no confusion.
Look at how Changes are classified and prioritised. If every Change is urgent, high priority, which one do you implement first? Classification is also really important – make your Change owners accountable for risk and impact assessments. If your company or Change Management tool has a Risk calculator use it as it enables Change requestors to assign a tangible risk to the Change (it removes the “if in doubt click medium” behaviour type) if not, I have some templates that I can share.
Next up is our old friend, the Change Assessment stage. This is the initial check that the Change is reasonable and makes sense, a sanity check if you will. Sounds obvious but unfortunately you can’t teach common sense. On one memorable occasion, I was reviewing a list of Changes on my pre-CAB FSC and one jumped out at me. The Change was to move a business critical server from a secure Data Centre environment to the Server technician’s desk. The Change justification? “My little legs are getting tired from walking to the server room all the time”. Needless to say words were had and the Change was removed from the schedule (I just wish I’d taken a screen shot).
Things to bear in mind when assessing a Change are benefits, both technical and to the business. There’s no point in having the newest most gadgetastic server is the world if your end users don’t see any benefit from it. Really think about the risk involved with the proposed activity for example:
Number of people affected
Loss of productivity
Make sure you have clear assessment criteria for managing Changes. Some examples could include:
Pre implementation testing – how do we know this Change will go as planned?
Deployment plan – does it make sense, if other teams are involved are they aware and do we have contact details for them? Are there any dodgy areas where we might need check point calls or additional support to mitigate risk?
Post implementation verification – ok we’ve done the Change, how do we make sure everything is as it should be?
Back out plan – hope for the best but prepare for the worst. What happens if something goes wrong on the night? Do we fix on fail or roll back? Are the Change implementers empowered to make a decision or is escalation needed?
Impact to other environments – “who cares about other environments?” I hear you ask. Let Aunty Vawns tell you why it matters from personal experience. Once upon a time in a galaxy far, far away I worked for a large investment bank in the city. A code Change to one of the most business critical systems (the market data feed to our trade floors) took longer than expected so instead of updating both the production and DR environments, only the production environment was updated. The implementation team planned on updating the DR environment but got distracted with other operational priorities. Fast forward to 6 weeks later, a crisis hits the trading floor, the call is made to invoke DR but we couldn’t because our market data services were out of sync. Cue a hugely stressful 2 hours where the whole IT organisation and its mum desperately scrambled to find a fix and an estimated cost to the business of over $8 million. Lesson more definitely learned that day.
So we’ve raised our Change and sanity checked it to make sure what we’re planning on implementing is sensible and won’t, you know, set anything on fire. The next step is the Change Advisory Board or CAB. When setting up your CAB, make sure you have a clear Terms of Reference statement which will give attendees a steer on how to prepare, good meeting behaviours and how to represent Changes effectively.
Not every Change has to go to CAB. In fact, I’d say you should use CAB for your big, complicated Changes that would have a major impact on the business. Monthly, BAU server patching? That’s a candidate for a standard Change. Keep CAB simple and uncluttered with an agenda that deals with Changes that are high risk, major impacting or have lots of complicated detail. Make sure the right people turn up and that all areas are represented (don’t forget Security if you have any ISO 27001 or NGN regulatory requirements).
An effective CAB agenda could look something like this:
Review of implemented Changes
Incidents (or if time is short Major Incidents) caused by Change
Forward schedule of Change
Candidates for templates / standard Changes
Improvements / CSI
Good news stories
Remember, CABs don’t have to be a 2 hour meeting where everyone is locked in a conference room. Look at Agile or Lean on how you can make efficiency savings, consider virtual Cabs where you can approve most things via your toolset and make full use of technology such as conference calls, WebEx’s or Skype.
The next stage of the process is Build & Test. The first thing to look at is developing standard build methods. Use automation where appropriate, it saves duplication work, reduces the probability of human area and the economies of scale can be huge. If automation is too expensive, ensure build methods are well documented and templated where possible.
Look at your testing environments and ensure they are fit for purpose. Every organisation will have different needs but when looking at this – do you have enough to cover your operational work eg a live environment for production work, a DR environment in case the worst happens and an environment dedicated to testing and training? If you are limited do you have a booking system in place so that testing / training / dev time can be allocated fairly?
What about Changes to non production environments? Are they covered by your Change process? Is there a monthly environment refresh just in case?
How do you test Changes before go live. Is it Bob from the Server team saying – “this’ll do” or so you have something more formal in place. Are the business willing to help support with Change testing? Getting the business to support testing can have huge benefits. One client I worked with has huge challenges with keeping the Marketing department happy with Changes that they had requested to the company website. The Marketing team didn’t provide any post Change validation and the amount of backed out Changes to the website or emergency Changes to fix errors such as typos were common place. By engaging with the Marketing team and asking for someone to be available for a set 20 minute period of the Change window (we called it the smoke testing phase) our effectiveness increased and so did our customer engagement.
When implementing Changes it’s not just a case of hitting a bit red button and shouting “fly my pretties, fly” to an imaginary army of flying monkeys (although that would be so cool). A professional approach and great communication are key to making this stage a success.
First up, your Change Schedule (aka your Forward Schedule of Change or FSC). Make sure it’s easily accessible to your stakeholders so stored somewhere central that’s easy to navigate. Make sure it’s dynamic and relevant – otherwise you risk your data being obsolete within minutes of hitting the print button. Make sure it’s fit for purpose, states what Changes are being carried out, when and by which Service. You don’t need expensive toolsets for this; you can do it in Excel.
Something that goes hand in hand with your FSC is your Projected Service Availability or PSA. Yes, I know ITIL 3 insists on calling it a PSO (Projected Service Outage) but lets not frighten the horses. Planned Service Availability implies we’ve got this. Everything is planned, tested and safe; you have nothing to worry about. In contrast PSO just says downtime. What do our customers not like? Downtime – no matter how well planned out. Again, Excel can be your friend here, keep it simple, a list of your Services – green for those that are up and running, amber for those due some maintenance time.
When looking at Change windows ensure they have been agreed in advance with the business and that they are codified as maintenance windows on any SLAs. Try and negotiate pre-approved Change slots where possible eg the third Thursday of every month between 22:00 and 01:00 we will be doing security patching. If you know in advance that a Change needs to take place outside the usual Change window – ask nicely! If you’re really good you might be able to get some SLAs relaxed so that as an organisation, you’re not penalised for carrying out break / fix work.
Finally, we have the Review stage. We’ve done our Change and everything has (hopefully) gone to plan. So let’s carry out a review of our implemented Changes. When things go well, brilliant! You have new candidates for standard Changes, work instructions or templates. If things go badly or you really do end up setting something on fire then let’s look at how we can do better next time. Carry out a Change review to look at what happened, what went wrong, what was the root cause, how did we fix it and how do we stop it from happening again? Involve Incident & Problem Management here as they have super powers in these areas. Above all you want your lessons learned to be captured, discussed and acted on. This could be a regular agenda at CAB meetings or form part of a Service Improvement Plan or SIP.
When reviewing Changes look at the benefits. In terms of business benefits, did we achieve the expected results? Have we got any customer feedback? We also need to look at the technical benefits i.e. have we increased stability, added resilience or fixed any Known Errors. If we have – brilliant – tell the Service desk so they can let our customers know.
I’d like to finish by saying that Change Management really is a critical process for managing controlling and protecting your live environment. As Change Managers we get to ask all the hard or awkward questions but on the bright side, we’re usually the first to know about cool new projects or sparkly new gadgets. Keep smiling, it will be fine. And if it isn’t, tweet me, I’m always happy to help!