Following on from Part 1 of our article on how to do Release Management – here are some tips on Release acceptance, rollout planning and communication & training.
Any development code that results in a change to the live environment should be under the control of Change Management and be managed by the Release Management process. If you are finding that a high number of Release RFC’s are being rejected at CAB meetings then you need to investigate the reason for this. All rejected Release RFC’s should be tracked and reported on by Change Management.
The Release should be tested in a controlled test environment with known hardware and software configurations. The current list of release acceptance criteria should be reviewed and updated if appropriate. Release acceptance criteria will differ widely for different organisations as different companies will have different requirements. Your list of release acceptance criteria could include the following:
- Has the release performed as expected across all development and test environments?
- Has the back out plan been tested successfully
- Are the release deployment instructions correct?
- Has all appropriate support documentation been updated?
- Has a training schedule been created / have all relevant teams received adequate training?
- Has the release undergone extensive User Acceptance Testing (UAT) with involvement from all impacted customer / user groups?
- Is the release performing as expected in test environments and free from defects?
- If testing has identified any defects with the release is it possible to deploy with a workaround or should the release be postponed?
- Have all impacted support teams received training in any new support functionality resulting from the release?
I like to borrow the Quality Gate concept from Six Sigma to make Release acceptance as effective as possible.Put simply, quality gates are a way of enabling your Release to move through the process quickly and safely making sure that all the quality criteria have been met. Quality Gates are not, I repeat NOT, road blocks or red tape; if anything they speed up the process (some checks can be automated) and cycle time is reduced because we’re getting it right the first time. Quality Gates are a set of predefined quality criteria that a software development project must meet in order to proceed from one stage of its lifecycle to the next and ensure. Quality Gates ensure that formal checklists are used throughout the life of a project so nothing can be lost, missed or ignored and that formal sign-off and acceptance occurs at each gate ie no nasty surprises.
Some examples of quality gates include:
- Code review: The senior developer will look at acceptable coding techniques and adherence to IT standards and best practices. The software design will be verified against the coding to identify any coding errors. Upon successful completion of the code review, the reviewing party will highlight required changes and corrections to the Release Manager so the appropriate action can be taken..
- Environment review; the test manager will look at the environments used for testing the Release to ensure they are fit for purpose, managed effectively and are being refreshed at pre agreed intervals.
- Ops review: The project manager will work with the support teams to review all support tasks needed to support the development environment and ensure all appropriate work instructions are in place..
- Sponsor review: The project manager will review the overall project performance with the project / release sponsor. This review will determine the status of project costs and schedule.
- Test review: The test lead and representatives from the Quality team will attend the test review to see if all builds were tested correctly and make sure the appropriate test scripts and procedures were followed.
- Deployment review: The Release Manager sits down with the relevant dev and support teams to run through the implementation plan and to ensure everyone is comfortable.
- Defect review; The Release Manager meets with business representatives, the Service Desk and the Project Sponsor to make a decision on if in the event of any defects; the Release is delayed or installed with Known Errors and workarounds.
There are lots of ways to deploy a Release safely into your environment. There is no one size fits all, it depends on the size and complexity of your organisation as well as appetite for risk. In the immortal words of Optimus Prime: “Autobots roll out!”
- Big Bang
- Phased / Pilot approach
Review the Release plan to include the exact details of the Release and how it will be executed. The release approach should also be considered to ensure it is appropriate of the type of release; different approaches such as “Big Bang”, phased / pilot, or parallel approaches can all be useful for different types of releases. A big bang release is whereby the release is deployed to all recipients at the same time. Advantages and disadvantages of the big bang approach are summarised in the following table:
Advantages and disadvantages of the Big Bang deployment method:
|Release is deployed to all users at the same time||More risky than other deployment methods because if the Release fails or causes a Major Incident the Release must be backed out for all users|
|Training is only required for the new system and not for running both the old and new systems in parallel||Training must be scheduled for all stakeholders prior to the release adding additional pressure to key release personnel|
|There is one deployment date which has been communicated to all stakeholders preventing any confusion around release scheduling||As there is only one deployment date, any delay to the release may cause adverse impact to certain departments|
Phased or pilot releases can be used to introduce new functionality to the end user base in a scheduled approach ensuring that the release has been successful at each stage before moving on to the next. Advantages and disadvantages of the phased / pilot approach are summarised in the following:
Advantages and disadvantages of the Phased / Pilot deployment method:
|Less risky than “big bang” deployment as the release is deployed to a set group of users at a time, thereby if the release fails it is backed out from one or a small number of user groups rather than the whole organisation.||Release implementation will take longer as it will be deployed in a staged manner rather all at once.|
|May enhance the relationship between IT and the selected pilot groups as the two will work very close together if a pilot approach is used.||Support for the release could be more expensive due to longer implementation windows eg needing contractors / consultants for longer|
A parallel approach can be used to reduce risk for business critical releases. A parallel release works by having both the old and new system run simultaneously for some period of time after which, if the criteria for the new system are met, the old system is disabled. Advantages and disadvantages of the parallel approach are summarised in the following table:
Advantages and disadvantages of the Parallel deployment method:
|Less risky than “big bang” deployment as the release as the original configuration is still available to users||Expensive as it involves running two versions of a system in parallel for a length of time requiring appropriate support personnel, licensing costs and system capacity.|
|Less risky than phased / pilot releases as you have an instant roll back in that your original service is still available||Risk of confusion to the user base as both systems are available at the same time|
|Additional training may be required for running both the old and new systems in parallel|
Push deployments are used where the service component is deployed from the centre and pushed out to target locations.
Advantages and disadvantages of the Push deployment method:
|IT are in control of when the Release is deployed and to which user groups||End users could be inconvenienced if the update is pushed during an important task|
|Can be automated or built into a Microsoft group policy||The network could experience performance issues if too big an update is pushed out|
|Ideal for critical security patches or antivirus updates|
A Pull deployment approach is used for software releases where the software is made available in a central location but users are free to pull the software down to their own location at a time of their choosing. As some users will never pull a release it may be appropriate to allow a pull within a specified time limit and if this is exceeded a push will be forced, e.g. for an antivirus update.
Advantages and disadvantages of the Pull deployment method:
|Users can schedule updates at a time that best suits them||Some users will never “get round” to installing the software so a combined Push / Pull approach should be considered eg users can pull at their convenience but If this hasn’t been done in x number of days the software is push out to the CI|
|IT doesn’t become a bottleneck; clients contact the server independently of each other, so the system as a whole is more scalable than a ‘push’ system||Scalability can become a bottleneck; unless you deploy several master servers and keep them in sync, that one master will start getting swamped as you add more and more clients and thus will become your bottleneck|
|Follows the self service model – end users feel empowered|
Communication & Training
It’s important to ensure that an appropriate level of governance is in place to support your Release Management is the introduction of governance. Setting up governance around a Release Management process will ensure the releases are implemented to a higher quality; it’s not just a case of channeling your inner Mr Burns (although that would be pretty cool).
A Release Board should be set up to control the formulation and implementation of Release strategy and in this way ensure the fusion of business and IT. The Release Board will also be responsible for managing risk, testing and formal senior management sign off in addition to the CAB approval discussed in the previous section. The Release Board will often make the final Go / No Go decision on whether or not the release can go ahead if a last minute defect is found.
The Release Board should meet frequently and should be made up of some of the following:
- Release Manager
- Change Manager
- Configuration Manager
- Project Office / Manager
- IT Senior Management
- Business Senior Management
Governance paths should be set up to underpin the Release Management process and establish guidelines, an example of which could be no first of type releases can be deployed by a big bang deployment and all impacted department should have additional floor walker support.
Regular release check point meetings should be held so that all stakeholders of the release are aware of the implementation details. Examples of meetings include:
- Pre release implementation checkpoint meeting
- Post implementation meeting
- Monthly process review meetings with stakeholders in the Release process to review the Release schedule, any issues, SIP, etc.
A release / service readiness review should be carried out to ensure the production environment is ready for the release. Such a review could be carried out in conjunction with Capacity Management to ensure that all capacity issues are tracked and addressed in the Capacity Plan.
The release implementation plan, back out plan and any associated technical documentation should be distributed to all stakeholders well in advance of the go live date to prevent any confusion. A communication to end users / customers should be issued via the Service Desk if a Release will have a noticeable effect on end users; it is useful to build up a bank of template Release e-mails so Releases are communicated in a standardised way.
Regular meetings should be held with Problem Management so that any Known Errors can be distributed to impacted stakeholders prior to the Release going live. There should also be a documented procedure for providing the Service Desk with a list of Known Errors before go live so that they are able to log incidents appropriately and relate them to the correct Known Error. Before any release is implemented, the Service Desk should receive training on any key changes to existing functionality and should be provided with “quick fixes” from the support teams to ensure that simple issues can be fixed by at the first point of contact where possible. New codes and templates for the release should also be set up in the Service Desk tool.
Join us for our second live ITSM webinar on Release Management on Thursday 25th February at 2:00PM GMT. You can watch live, or on demand by registering here.
That’s all for now, come back soon for our final article in the series where we’ll look at go live, early life support and review & close.