Is there a lab exam in your future? According to recent comments from Dan Truax, of Microsoft's certification skills and assessment-strategy group, the answer is probably not. But you should expect some "exam innovations" in the area of hands-on testing. "We have no specific plans to launch a lab exam," Truax said in a recent MCP Magazine article. Instead, he told MCP Mag that Microsoft intends to incorporate simulations within exams, and that it's possible that future exams will include more product interaction.
These seemingly innocuous remarks touch on a deep debate within the certification community right now, and cogently state Microsoft's position on the subject. On the one hand, there are some certification -- and exam -- designers who strongly believe in performance-based testing where individuals are faced with real-world scenarios and let loose in a lab environment to demonstrate their analytical, troubleshooting and problem-solving skills. On the other hand, most certification -- and exam -- designers build their programs to test candidates' knowledge and skills mastery within the framework of a typical computer-based testing center like those run by Prosoft or VUE.
There's an interesting trade-off at work here: Lab exams do indeed provide a more realistic measure of candidates real-world skills and knowledge, but they also require special testing centers with dedicated equipment, staff and a heavy investment in lab design, set-up, and management.
The Cisco CCIE certification, for example, gets much of its luster and cachet from the one-day compulsory lab exam that is regarded by many in the IT industry as among the most difficult of all certification exams. There are NO paper CCIEs, period. But then, too, there are less than a dozen lab examination centers for the CCIE worldwide, and candidates must invariably travel to those centers to be tested, thereby incurring travel costs as well as a hefty $1,250 examination fee. My sources tell me that less than 50% of candidates pass on the first try, and many take three tries to pass, so the expense factor gets pretty serious, pretty fast.
On the other hand companies and organizations -- like Microsoft -- that want to reach the largest possible global audience build their exams to fit within the capabilities of ordinary computer-based testing centers. When their exams include problem-solving, system configuration or product interaction, they invariably rely on simulations that will work within the limitations of a typical testing center PC -- which are often less than state-of-the-art in their capabilities, and may not even be able to run the actual software or products on which candidates are tested.
But don't be too inclined to dismiss simulations outright as inferior substitutes for the "real thing." After all, if airline pilots can learn to fly aircraft worth hundreds of millions of dollars -- or more -- by simulation, surely IT professionals can be properly tested in a simulation environment?
The key here is to make sure that simulations match the real thing as much as possible. Even more important, the subjects or activities that get simulated must match real-world knowledge, skills and job requirements as much as possible, or the relationship between what's tested and what's needed in the workplace grows increasingly tenuous. Alas, despite Microsoft's many good efforts to improve their exams and their relevance to real-world concerns, there are still plenty of "paper MCSEs" in the marketplace.
But by building better simulators, and covering more key system interactions, activities and workaday tasks, Microsoft and other testing center adherents can improve the match-up between the "testing world" and the "real world." Truax's comments seem to indicate that Microsoft is aware of the issues and intends to keep improving its track record. It will be interesting to watch how this dialog unfolds further as a new generation of Windows Server.NET and other related exams emerges in the next year or so. I expect Microsoft to beef up its simulation capabilities considerably and to model increasingly more realistic and difficult administrative and management tasks as it keeps trying to improve its exams and the certifications they warrant.
Ed Tittel runs a content development company in Austin, Texas, and is the creator of the Exam Cram series. He's worked on many books on Microsoft, CompTIA, CIW, Sun/Java and security certifications. His team is currently at work on TICSA and Security+ study guides for Que Certification Press.
