Studies have shown us that a large majority of IT projects fail either in part or outright for non-technical reasons that could have been prevented with proper planning.
Think about the last release of a major new or changed service you put into production that exceeded the time frame or budget allotted for it or otherwise failed to deliver on expectations.
Are you deploying Windows Vista right now? How is that going? Is it really a technical issue that's holding it up or is it a failure to understand all the requirements of the organization?
IT managers can reduce these problems by implementing the Release Management component of the Information Technology Infrastructure Library, known as ITIL. ITIL is a management framework that contains processes created from best practices that IT managers can use in their own organizations.
The Release Management process is one ITIL concept that you need to approach from a business angle. To determine the approach, an IT shop needs to identify what the current state of the Release Management process is, what the desired future state of the process should be based on ITIL, and what gaps exist.
To prioritize improvements, IT departments have to understand what is causing the most pain and then review their options in terms of what resources are available to deal with the problems. Other areas to consider when deciding how to rectify a problem include time frame, costs, current process maturity and the ability of the organization to change its culture.
Once the Release Management process begins to take shape, develop a release policy to identify a base set of requirements that will apply to releases it governs. Some groups will do one enterprise release policy while others will write a policy per service -- such as a release policy for enterprise Exchange Servers, for example.
A release policy sets standards around how a release into production is to be managed to ensure that the requirements of the business are met, tested and implemented. Areas to consider include what minimum testing is required, who does it and when. The idea is to remove ambiguity around what is to be done and who is to do it.
The following are potential elements to be covered in a release policy:
- Release units – This identifies what components -- such as hardware, software or documentation -- constitutes a release. This can vary from service to service. Some services may be hardware plus the Windows Server operating system plus configurations stored in documents to make it ready to be a file server. Others may include hardware, Windows Server, SQL Server and necessary configurations stored in a configuration utility.
- Release identification – How releases will be marked is a point for standardization. Each group has its own methods, and many follow something akin to major.minor.revision to render sequences such as "version 1.2.3." But if it is major release 1, then the minor release level is 2 and the revision level is 3.
- Types of releases – The release policy can outline what will go into "full" releases, where everything is updated. The policy will also outline the schedule for those releases and for "delta" releases, which are incremental updates. The last type of release is a "package," which is a combination of full and delta releases into one release type. These are handy when you want to release a full application with a delta service pack at the same time.
- Definitive Software Library – The policy may outline the physical and logical location of the DSL plus spell out how the DSL is to be managed. The DSL is where all authorized software for production is checked in, and it is the only place that a release engineer creating builds should draw software from. This ensures that only legal and approved software is used in the release process, and it reduces the chances of malware and unlicensed software from entering production.
- Definitive Hardware Store – Again, the policy may outline the location of the DHS, which spares to maintain and what the procedures are for checking out hardware. The idea behind the DHS is to enable the organization to have ready access to spare hardware that meets organizational standards on a moment's notice.
- Testing and acceptance – The policy must identify the minimum testing protocols needed prior to a release that are deemed acceptable for production. A critical service may require formal unit, integration, capacity, security and operational testing, for instance, with test results stored for one year. The policy may mandate that all releases be tested by Qualys in the test environment before production deployment to identify security vulnerabilities. It must also identify who has the final ability to approve a release for production.
- Backout plans – What are the acceptable backout plans if a release into production fails during implementation? The policy must also identify this critical element. If a release fails, there must be a reliable means to restore the service into production efficiently and effectively.
- Rollout planning – It's not uncommon for groups to skip important phases, such as a requirements definition. The policy must specify what level of rollout planning is required with what phases and tasks to sufficiently manage the creation, testing and deployment of the release.
- Communication and training – The policy must identify what communication and training are required prior to releases going into production as well as what needs to be done afterward.
- Distribution and installation – The approved distribution and installation tools and procedures must be identified in the release policy. Automation is good for improving efficiency and effectiveness and reducing human error, provided there are solid processes that the tools are supporting. Using MOM is great if there is a process safeguarding the organization from the automated distribution of 10,000 changes that contain a virus.
For IT managers struggling with how to ensure quality in their releases, having a standard release process with defined policies, standards and procedures will help reduce errors.
By creating a release policy, you can clarify all the roles and responsibilities between development, procurement, project management, security and data center operations. That way, you can achieve a more stable environment with services that meet the needs of the organization -- not to mention happier customers and end users who benefit from having services they can count on.George Spafford is a principal consultant with Pepperweed Consulting and helps IT organizations in the areas of technology and process implementation. Spafford is an experienced practitioner in many aspects of IT operations, including strategy, audit and operations. He is a speaker on topics that range from security and IT governance to IT process improvement. He is the co-author of The Visible Ops Handbook.
This was first published in October 2007