For problem management teams and even for security management teams, the first question they should always ask is "what changed?" People with malicious intent aren't going to bother issuing a request for change and neither will someone who makes an honest mistake. The only authoritative sources for all changes are detective controls.
Change management is essentially a risk-management process. In it, the risks of making a change are balanced with the risks of not making a change. Although an organization must be allowed to attain its goals, IT managers need to minimize risks associated with changes to production services.
Using detective controls is one way to reduce any impacts that might occur from changes. Imagine a critical IIS service that enables supply chain management and ties in to Microsoft SQL Server. If a change, such as a Windows operating system patch, introduces errors to that system -- perhaps causing it to crash -- what will the impacts to the business be? They might range from losing reporting capabilities to having a distribution center idle because orders from partners cannot be received.
What is a detective control?
A detective control is a procedure, possibly aided by automation, that reviews historical events. A change management detective control could be a procedure with a tool that compares the current configuration of a production configuration item to an approved stated and then generates a report of discrepancies. For example, monitoring the files in the system32 folder and reporting on new, changed or deleted files compared to the approved golden build is a detective control.
A detective control can be an entirely manual procedure accomplished by one or more persons. Using a manual control, which means that human intervention is required, is problematic for change detection because:
In contrast, using automation to monitor and report findings that are then reviewed by an objective party is far more realistic . An automated control, such as Tripwire Enterprise, is faster, more consistent and cost-effective. With the automated control, it becomes possible to run tasks that search for changes each night or even after each shift.
The work is slow and tedious, and there is a high risk of overlooking critical changes. People are prone to human error and, thus, reviews are likely to be inconsistent. With a single Windows server potentially having tens of thousands of files and thousands of registry entries, manual review on a timely and cost-effective basis is unrealistic.
One important note concerning automated change detection tools: Have a carefully defined scope both in terms of the configuration items that you will monitor and the elements you will monitor on each.
If you choose to monitor every file and every registry key on a Windows server, then the reports will be voluminous and will quickly lose meaning because of all the changes to temporary files, data files and so on. Instead, start by focusing on the critical parts of each configuration item that are important -- for example, the system32 folder and a couple of key application folders. That will better ensure that the reports are providing change information relevant to what matters and not just nervous chatter.
Reaping the benefits of detective control
In organizations that have not traditionally had the discipline of change management, convincing the culture to change and follow a new, or even the existing, process can be difficult. It is typically quite easy for parties to bypass the process and make a registry change, update files, etc.
One of the biggest benefits of a detective control is that it identifies and reports on all changes -- whether users followed the defined process or not. IT managers can then work with those users who choose to circumvent the process and get them instead to follow the process. It quickly becomes apparent to users that changes are detected and that there are defined consequences for not following the process.
A new application is -- or should be -- designed with regulatory compliance requirements in mind, and it should be properly tested prior to rollout into production. Or, where there are legacy applications, baseline testing might have been performed to validate that the system does function as intended. In either case, there is a point in time where stakeholders, including auditors, could substantiate with a high degree of certainty that the system is in compliance.
With uncontrolled changes -- meaning changes that do not follow change management procedures -- you lose your ability to guarantee that the system is in compliance. A DLL might have been replaced or an EXE upgraded and the new files contain flaws that subvert expected outcomes.
If changes aren't controlled, then risks aren't controlled. If that happens, an organization might find itself out of compliance and could be required to pay fines and take costly corrective action.
Automated change management detective controls can bring operational, security and regulatory benefits to organizations. All changes should follow an approved process. By detecting the ones that do not, IT managers can take corrective action that lets them properly manage risks to the organization.
George Spafford is a principal consultant with Pepperweed Consulting and helps IT organizations in the areas of technology and process implementation. Spafford is an experienced practitioner in many aspects of IT operations, including strategy, audit and operations. He is a speaker on topics that range from security and IT governance to IT process improvement. He is the co-author of The Visible Ops Handbook.
This was first published in September 2007