Companies administering high-stakes exams are required to run pilot tests in order to gather the statistics they need to build reliable and valid exams. Typically, IT industry companies require test development on a timeline that is tighter than that required for other licensure or certification organizations. Test development consulting firm The Donath Group was approached by one of their clients, a major IT company that required their exams to be ready in time for a conference. This meant that the exams needed to be developed even more quickly than usual.

While test development for IT firms usually takes a minimum of four to six months, The Donath Group established a rapid test development approach to expedite the process. This typically takes 12 weeks. However, because of the client's targeted release date, The Donath Group had to consider approaches to complete the test development in an even shorter timeframe than their customary rapid test development timeline.

Beta tests delivered by standard methods, going through a proctored test center environment, require a minimum of 3 weeks to build and administer the exam. So The Donath Group had to take a completely new approach to meet the company's minimal timetable. While it was essential to test a significant number of people in order to generate adequate statistics, the time to market was equally critical.

The Donath Group proposed an innovative approach to controlled, non-proctored pilot testing: hosting online Perception tests in a non-proctored environment. The company agreed to let The Donath Group build a prototype of this new approach. Administering the beta tests with controls on all browser functions, the consultants were able to ensure that test items could not be copied, printed or saved.

Candidates could log on from any location within a narrow time frame and could take the exam only once. Without proctors, it wasn't possible for people to get certified by passing the beta test, as would normally be the case. However, by taking the beta test they would be able to anticipate the topics that would be covered in the final certification exam. Candidates benefited not only by learning their strengths and weaknesses, but they were also offered discounted fees to take the final, published exam.

The Donath Group hosted the pilot tests and monitored them closely. More recently, they administered a subsequent pilot test for the same client. Of the 400 people who registered for that exam, nearly all of them participated over a period of three days. This compares to a normal timetable of two to four weeks for the actual testing, often with a far smaller sample than 400. Not only was the number of participants high: the breadth of global participation was much greater than would normally be the case.

The Donath Group implemented specific approaches to facilitate implementation by establishing a beta registration process and developing a utility to help load and export test items into Perception.

Given the importance of the beta test, they also took some important steps to maintain the integrity of the process, such as building multiple test forms to minimize test item exposure.

The Donath Group met the client's deadline, and the pilot tests ran very smoothly. The consultants pulled in the test results and conducted an in-depth statistical analysis in order to build equated test forms. They also set the cut-score determination for the actual certification exam. The client was extremely pleased with the process, which reduced administration costs and made it possible to publish final certification exams far sooner than would have been possible otherwise. Since then, the company has asked The Donath Group to plan and host more beta tests using Perception. It's likely the consultants will utilize Perception Secure Browser to further ensure the security of the pilot tests.

The Donath Group had been concerned about proposing such a radical change in the process of running pilot tests, simply because it was such a departure from the usual model. However, the efficiency of the system as well as its cost-effectiveness made for a win-win situation for the company as well as its certification candidates.

IT professionals are very excited about the prospect of participating without having to make a special trip to take a beta exam. For global organizations like this client, the costs of administering beta exams in some countries can be very high, as are costs for candidates. As a result, some candidates may be reluctant to travel and pay to participate in a beta test. Administering pilot tests online offers a low-cost solution that makes it easier for people to participate. This solution makes pilot testing truly worldwide, which is a big help when it comes to analyzing data and ensuring that it is truly representative of the global certification audience.