But for repeated use, automation is usually quicker and more accurate.

For instance, NASA uses Empirix’s e-Test Suite and FarSite testing service for regression and performance testing on its Web-based grant management application. “There are always new rules and requirements, so this is not an application that we’ll finish and never touch again,” explains Idriss Mekrez, senior software quality architect for NASA’s Peer Review Services. “Automated testing allows us to define the basic functionality the system needs to deliver and test those processes automatically.”

For mission-critical applications, automated testing is usually well worth the effort, given the consequences of failure. “The applications that most need testing are those with the biggest impact on the business,” says Dan Koloski, product manager at Empirix. “If an application is cus-tomer-facing and revenue-generat-ing, it is ripe for substantial testing.”

While functional testing tools typically cost $5,000 to $10,000, performance tools can cost $50,000 to $200,000. Renting a performance tool is less expensive—around $10,000 to $20,000 for a couple of weeks, according to Povlot. Some tool vendors rent their products, as do testing consultants.

Open-source tools are also available. “They’re not as polished as commercial tools, and you have to be a programmer to use them,” says Elisabeth Hendrickson, president of Quality Tree Software Inc., a testing consultancy in Pleasanton, Calif. “But anyone doing testing ought to know how to code anyway.”

Open-source tools aren’t as mature as commercial products, warns Danny Faught, owner of Tejas Software Consulting in Fort Worth, Texas. However, he says, “if somebody has programming expertise, they should definitely consider open source. If it’s going to save you $100,000 or more, it’s worth a little work upfront.”

choosing a tool Regardless of whether you rent or buy, here are three key factors on which to evaluate any tool:

“Our applications pass
data like treatment
and patient
information, so there
is a real-time flow
between them.
We could only test
the whole system
effectively with
automation.”

AsHisH KAtrEKAr,
quALit Y AssurAncE MAnAgEr,
vAriAn MEDicAL s YstEMs inc.

PrOtOcOL suPPOrt

Not all tools support every platform or protocol. Some support a range of protocols but charge extra for each one. As Povlot explains, “Most [of the tools] handle basic Web applications, but the selection narrows when you look at the different operating systems, databases and interfaces that are included.”

That was a deciding factor for Time Warner Inc. when it evaluated performance testing tools. “With most of the tools, if you wanted to test under SSL, ODBC and Oracle, you have to buy those components separately,” says Robert Caruso, senior business systems analyst at Time Warner in New York. The company decided on Compuware’s QACenter Performance Edition, which bundled all of the required protocols into its basic suite.

Protocol support was also a consideration for Apollo Group Inc., parent company of Phoenix University. “We were testing EJBs, RMI [Java’s Remote Method Invocation], Oracle, PeopleSoft, [BEA Systems’] Tuxedo transaction monitor, and Web applications,” says David Pinkus, senior director of software at Apollo Group. Pinkus opted to go with Mercury’s tool and pay extra for the protocols because he felt that “they had a richer history with more diverse protocols.”

 

MAintAinABiLitY The ability to easily update scripts when the tested application’s user interface changes is a concern.

“Maintainability is a huge issue,” says Quality Tree’s Hendrickson. “What the high-end tools do is provide a GUI map, or test frame, which is an abstract layer between fields [on the interface] and the test script, so that if the name of a field is changed, then you only have to make that change in one place and all of the scripts will work. They centralize the references to the user interface in one place.”

Componetization of scripts also helps. If one part of a script, such as the log-on process, changes, only the component dealing with that process needs updating, not the entire script. Mercury says that in its products, that’s part of the “business process testing” feature for creating application workflows by assembling the process components. It also enables nontechnical managers to change a test case or create a new one.

“When a business process changes, the business analyst knows immediately, whereas a QA engineer may not. Now the analyst can just create a new test case,” says Rajesh Radhakrishnan, senior director of product marketing at Mercury.

cOMPAtiBiLit Y WitH

YOur APPLicAtiOn

Just because the software supports your protocols, that doesn’t guarantee it will work with the software you need to test. Hendrickson recommends bringing competing tools in-house for trial tests with your applications. “Often, one tool will talk to a particular application better than another. That’s why I advise a hands-on evaluation,” she says. “Give it a small set of representative tests you plan to run.”

That also allows you to see how easy the product is to use. When Time Warner evaluated performance testing tools, it invited vendors to demo products using Time Warner’s applications. “We wanted to see whether we could take it out of the box and get it working,” says Caruso. “We don’t have a dedicated QA staff, so we needed something our developers could bolt onto their development process.”

A trial also lets you see if the
product lives up to its marketing
claims. When the Apollo Group