In order to secure an important sale of an HR application to a major global publisher, ProAct needed to deliver a proof-of-concept to their prospect. They had to prove their application would work for the publishing house's specific environment before they would buy it. So testing the solution's configuration and use for the publisher before going in to do the proof-of-concept was critical to the win.
ProAct needed pre-production load testing done on an array of transaction types within the application; and they needed the testing done fast, but lacked the in-house expertise and manpower to do the work. ProAct turned to Rational Test Bureau partner RTTS for help. RTTS, a New York City based IT application and architecture testing firm, has distinguished itself as a top provider of automated testing for Fortune 500 corporations and had already proven itself an effective testing service on many previous engagements with ProAct. RTTS' world-class level of testing expertise and knowledge made them a welcome candidate for participation in the Rational Test Bureau program.
Rational Test Bureau partners deliver access to Rational's solution for functional, performance, reliability testing and runtime analysis, Rational Suite™ TestStudioÂ®, combined with the expert testing services of our partners. Businesses and organizations using the Test Bureau don't have to purchase, install, or learn how to implement software tests. Many companies whose software is their livelihood find the services advantageous when faced with urgent projects coupled with resource constraints, even if they have in-house testing capabilities.
The Prospect's Needs
ProAct was in exactly that position of having an urgent project and resource constraints when their Vice President of Professional Services, Carlo DeMarco, went to RTTS for assistance. The scope of the testing included both load and functional testing, to ensure a flawless proof-of-concept for the publishing house.
ProAct's software for the publishing house had to meet these performance standards:
-Handle up to 500 concurrent users
-Fully load any one page within 21 seconds
-Fully load any single object on any page within 10 seconds.
-Perform up to 5,000 page requests per hour
-Editing and updating employee mailing information
-Updating and retrieving employee paycheck information,
-Retrieving against a logic expression for medical benefit information
-Accessing 401k information from an external site
-Proving the single sign-on capability via Netegrity's SiteMinderÂ® worked with the application
Rational Test Bureau Partner: RTTS
RTTS is a professional services organization that specializes in the testing of IT applications and architecture. The company has offices in New York and Boston, and has been serving Fortune 500 and mid-sized companies nationwide since 1996.
For RTTS project lead Jon Harris, Rational Suite TestStudio was the testing tool of choice for their client because it performs functional and load testing simultaneously and allows the virtual users to do more than one thing at once, the way people use applications in real life. The tool is highly scalable and creates runtime analysis and code-coverage metrics in the background, correlating bugs to the code causing them and letting the RTTS testers know what code was exercised during testing. This meant that Harris' RTTS team could test ProAct's business critical application with confidence in their ability to find any bugs that could keep ProAct from winning its sales engagement with the publishing house.
RTTS staffers conducted the tests in their lab using their own equipment, saving ProAct from assigning space, reallocating existing equipment, or purchasing additional equipment to run the tests. RTTS set up the load testing to ramp up to 500 concurrent users by using five machines, each running between 100 and 150 Virtual Testers. A virtual tester or virtual user emulates the complex actions of a person using an application.
The machines used were configured with PentiumÂ® III Processors, 1.2 gigabyte hard drives, .5 gigabyte of memory, LAN cards, and fiber optic T1 connections to the Internet. During the testing, RTTS used Rational Suite TestStudio's test management interface to continuously monitor the machines for adequate CPU and memory resources, and to ensure the T1 connection was not saturated. The testing configuration included third-party software that the publishing house uses, so that ProAct could guarantee their solution would function with it.
The Process and a Bonus Find
The tests simulated users logging-into the system and executing the various types of representative transactions. According to RTTS' Harris, Rational Suite TestStudio's graphic scheduler was a powerful asset in testing the transactions under varied user loads. "Using Rational Suite TestStudio's graphic scheduler, we were able to schedule users to run different transactions and different mixes of transactions to truly simulate real life," said Harris. "Most other products on the market today allow each virtual user to run only one script."
Throughout the testing process, RTTS testers recorded scripts of virtual user actions at critical points as they scaled-up in increments to 500 concurrent users. Between each test they verified that everything executed correctly, and that they had the correct volume of transactions. But then a problem occurred: something was getting in the way of scaling up to the maximum required performance level.
Thanks to the runtime analysis capability that Rational Suite TestStudio provides during testing, the RTTS testers were able to pinpoint the cause to an information-sharing problem between two third-party software components required for the proof-of-concept. The problem in the compatibility of the other vendors' products was keeping ProAct's application from scaling as required.
The RTTS testers solved the problem after they discovered a pattern of error messages occurring during login and after the first few pages of testing. By looking at response times and error messages, and after conducting still further tests with Rational Suite TestStudio on the third-party server software, RTTS was able to narrow the problem to the way one third-party application shared information with the required third-party server. After consulting on the project, the server software vendor issued a patch that resolved the problem, allowing the test of ProAct's application to immediately and successfully scale and be tested at 500 concurrent users within the assigned performance constraints.
Win, Win, Win
Rational Suite TestStudio performed as promised for RTTS and RTTS enabled ProAct to be confident of their application's performance during the proof-of-concept. ProAct in turn conducted a successful proof-of-concept, a critical accomplishment that motivated ProAct's client to purchase the application. It was a winning situation all-around, with particular benefits for ProAct. "We couldn't undertake this kind of testing on our own," said Carlo DeMarco, Vice President, Professional Services for ProAct Technologies. "RTTS and the Rational Test Bureau Program saved us valuable time, manpower, and resources and allowed us to concentrate on what we do best-develop world class enterprise-wide software for employee resource management."
"This scenario clearly illustrates the reasons we created the Rational Test Bureau," said Bill Chandler, Rational Software's Program Manager for the bureau. "Here you had ProAct, a company that realized the importance of testing, yet didn't have adequate expertise, manpower, or infrastructure to do it themselves. Additionally, they were up against a tight deadline and delivery of the proof-of-concept was critical to their winning a large order for their products. RTTS, like all of our Test Bureau Partners, was uniquely positioned to provide timely, cost-effective testing services using Rational's industry-leading testing platform."