Publication
: ST ES BE TIC eb OAs AC W d S PR ting an Tes ces rvi Se
A
VOLUME 3 • ISSUE 11 • NOVEMBER 2006 • $8.95 • www.stpmag.com
Using JUnit To Validate Public Java APIs
Metrics Can Help You Grab Testing’s Brass Ring
Security Zone: Tighten The Reins with Code Screening
Ride Automation SDLC Keep Development and Testing in Lockstep
Ship Software OnTime.
™
" / ÓääÈ The Fast & Scalable Team Solution for... Defect & Issue Tracking • Feature & Change Tracking • Task & To-do List Tracking • Helpdesk Ticket Tracking OnTime is the market-leading project, defect and feature management tool for agile software development and test teams. OnTime facilitates tracking, analyzing and trending team-based software development efforts in an intuitive and powerful user interface. A fully customizable UI, powerful workflow, process enforcements, two-way email communications and custom reports combine to help software development teams ship software on-time!
Available for Windows, Web & VS.NET 2003/2005 OnTime 2006 Professional Edition
OnTime 2006 Small Team Edition
• For Teams of 10 to 1,000 Members • From $149 Per User
• • • •
For Teams up to 10 Members Free Single-User Installations $495 for 5-Team Members $995 for 10-Team Members
800·653·0024 software for software development™
www.axosoft.com
Only $495 for up to 5 Users • Only $995 for up to 10 Users Free Single-User Installations
The days of
‘Play with it until it breaks’ are over!
Introducing: ®
TestTrack TCM The ultimate tool for test case planning, execution, and tracking. How can you ship with confidence if you don’t have the tools in place to document, repeat, and quantify your testing effort? The fact is, you can’t. TestTrack TCM can help.
Issue & Defect Management > Test Case Planning & Tracking Configuration Management Automated Functional Testing Essential Process Management
In TestTrack TCM you have the tool you need to write and manage thousands of test cases, select sets of tests to run against builds, and process the pass/fail results using your development workflow. With TestTrack TCM driving your QA process, you’ll know what has been tested, what hasn't, and how much effort remains to ship a quality product. Deliver with the confidence only achieved with a well-planned testing effort.
• Ensure all steps are executed, and in the same order, for more consistent testing. • Know instantly which test cases have been executed, what your coverage is, and how much testing remains. • Track test case execution times to calculate how much time is required to test your applications. • Streamline the QA > Fix > Re-test cycle by pushing test failures immediately into the defect management workflow. • Cost-effectively implement an auditable quality assurance program.
Download your FREE fully functional evaluation software now at www.seapine.com/stptcmr or call 1-888-683-6456. Managing Process, Change & Quality Throughout the Enterprise
©2006 Seapine Software, Inc. Seapine TestTrack and the Seapine ©2006 Seapine Software, Inc. Seapine TestTrack andare thetrademarks Seapine logo logo of Seapine Software, Inc. All Rights Reserved. are trademarks of Seapine Software, Inc. All Rights Reserved.
A MESSAGE FROM THE EDITOR
VOLUME 3 • ISSUE 11 • NOVEMBER 2006 EDITORIAL Editor Edward J. Correia +1-631-421-4158 x100 ecorreia@bzmedia.com Managing Editor Patricia Sarica psarica@bzmedia.com Copy Editor Laurie O’Connell loconnell@bzmedia.com
Contributing Editors Scott Barber sbarber@perftestplus.com Geoff Koch koch.geoff@gmail.com Editorial Director Alan Zeichick +1-650-359-4763 alan@bzmedia.com
ART & PRODUCTION Art Director LuAnn T. Palazzo lpalazzo@bzmedia.com
Art /Production Assistant Erin Broadhurst ebroadhurst@bzmedia.com
SALES & MARKETING Publisher
List Services
Ted Bahr +1-631-421-4158 x101 ted@bzmedia.com
Nyla Moshlak +1-631-421-4158 x124 nmoshlak@bzmedia.com
Advertising Sales Manager
Reprints
David Karp +1-631-421-4158 x102 dkarp@bzmedia.com
Lisa Abelson +1-516-379-7097 labelson@bzmedia.com
Advertising Traffic
Accounting
Phyllis Oakes +1-631-421-4158 x115 poakes@bzmedia.com
Viena Isaray +1-631-421-4158 x110 visaray@bzmedia.com
Marketing Manager
Marilyn Daly +1-631-421-4158 x118 mdaly@bzmedia.com
READER SERVICE Director of Circulation
Customer Service/
Agnes Vanek +1-631-421-4158 x111 avanek@bzmedia.com
Subscriptions
+1-847-763-9692 stpmag@halldata.com
Cover Photograph by Pete Corr
President Ted Bahr Executive Vice President Alan Zeichick
BZ Media LLC 7 High Street, Suite 407 Huntington, NY 11743 +1-631-421-4158 fax +1-631-421-4130 www.bzmedia.com info@bzmedia.com
Software Test & Performance (ISSN #1548-3460, USPS #78) is published 12 times a year by BZ Media LLC, 7 High Street, Suite 407, Huntington, NY 11743. Periodicals privileges pending at Huntington, NY and additional offices. POSTMASTER: Send address changes to BZ Media, 7 High Street, Suite 407, Huntington, NY 11743. Ride along is included. ©2006 BZ Media LLC. All rights reserved. Software Test & Performance is a registered trademark of BZ Media LLC.
NOVEMBER 2006
Brushing Up On My Latin Repetitive tasks are bad tains some excellent guideenough—having to perform lines for adopting automathem repetitively is insane. If tion best practices, includsomething can be automating the first of a three-part ed, it should be. Even if it series on Automation SDLC takes 10 times longer and by Bob Galen, author and a costs a hundred times more 25-year veteran of the softthan the original task itself, ware industry. Other beneit pays in the long run to fits of automated tests? automate your tests. They never get sick or go Consider this: A developon vacation, they never forEdward J. Correia er asks a fellow team memget, and they never, ever Editor ber if their manager, I’ll call complain of boredom. him Bob, was coming in that day. “Bob’s Welcome to Version 2.0, Beta 1 out sick,” was the reply, and no one knew That’s the label I’m putting on this issue, when he’d be back. my first as this publication’s editor, as I But without Bob, no one would be take over for Lindsey Vereen. able to finish writing those use cases and I remember the excitement when execute them by the end of the week, Lindsey was hired as editor of ST&P. He their deadline for meeting the release was quite a catch, we thought—joining us schedule. The developers grew frantic. shortly after his ascension to editorial On Thursday, Bob, still under the director of CMP’s Embedded Systems weather, dragged himself in to the office, Group—and his will be big shoes to fill knowing that he was the only one in the (and I don’t know much Latin). organization capable of completing the Lindsey’s last column contained a QA phase of the development project. kind introduction of me (and some He should not have been. Latin), and he mentioned our first meetEvery company has a Bob, someone ing, an industry dinner in March of 2002. who knows the processes inside and out Lindsey sat to my right, and Michael Barr, and is a seemingly indispensable part of then the editor of Embedded Systems any project. For developers without autoProgramming, sat to my left. I don’t recall mated test processes, scenarios like this being terribly insightful or knowledgecan be common, and critical steps of an able about the industry that night, as application’s quality are placed in the Lindsey so kindly described me, but I do hands of a single “expert,” left to chance recall an engaging chat with Michael or ignored all together. about a book he’d recently published, “We’ll fix it in version 1.1” or words to which I happened to have read. One that effect have likely been heard by adjective that entered my notes from that many of you. If quality is Job One, then evening described Lindsey as “funny.” test automation should be Job 1.1. Over the next several issues, you’ll see The benefits of test automation are some changes—some obvious, some subpalpable and easily calculated. They tle—that I hope you will agree make include more complete code coverage, ST&P livelier and easier to read. I also shorter execution time, reduced cost and plan to gauge your interest in past issues fewer errors in the final product. In and future topics through upcoming essence, automation delivers consistency reader surveys. In the meantime, please that can be measured and documented. send me your thoughts at ecorreia There are numerous studies to prove @bzmedia.com. these facts, some of which are touched Carpe Diem. ý on in these pages. This issue also conwww.stpmag.com •
7
collaborate automate accelerate
Manage your distributed test teams Organize all of your test assets in a single, shared repository Document your tests and results in a standard format for compliance
Get a grip on your testing with Worksoft Certify速 To view a demo on Worksoft Certify or download the Accelerating SAP: An Innovative Approach to Test Automation white paper, visit www.worksoft.com/stp1106 or call 1-866-836-1773. Worksoft and Worksoft Certify are registered trademarks of Worksoft, Inc.
VOLUME 3 • ISSUE 11 • NOVEMBER 2006
Contents
14
A
Publication
COV ER STORY
How to Build Tests And Apps In Tandem
Does manual testing have your team running in circles ? Get automated the smart way, by incorporating elements from the SDLC to organize and ramp up your automation effort. Part 1 of 3. By Bob Galen
20
Build Public APIs With Java Specs
Design public APIs using application domain modeling techniques and design patterns. By Anna Mikhailova
28
Automate App Tests to Achieve Maximum Benefit
Depar t ments
Tests are measured by code coverage, execution time and cost. Here’s how to squeeze every penny from all three. By Shaun Bradshaw
7 • Editorial Welcome to ST&P 2.0, beta 1! This month, automation takes center stage.
9 • Out of the Box
37
Scan Source Code For Security Gaps
Static analysis tools—and security awareness training—can help you put security first, throughout the entire development life cycle. By Joe Basirico
New products for developers and testers.
12 • Peak Performance After all these years‚ a miracle: A tool maker finally listens! By Scott Barber
43 • Best Practices What would an all-services, all-the-time future be like for testers? By Geoff Koch
46 • Future Test SOAs are notoriously untestable—but you can do it with a highly structured By Chris Benedetto approach. NOVEMBER 2006
www.stpmag.com •
5
Out of t he Box
On a Quest To Test Any Remote App TestQuest (www.testquest.com) at the end of this month is set to begin shipping CountDown, a .NET-based platform that it claims can launch and control real tests on any device running any operating system anywhere on the Internet. According to Martin Hahn, CEO of the Minneapolis-based startup, the tool can prevent enterprises from losing developers to travel time. “Rather than having to fly people back and forth or shipping a device [from] India, you can set up the connection over the Internet and see what’s happening in your office.” The CountDown platform is built around agents, device-specific utilities run on and control myriad target devices including desktop or laptop computers running Linux, Mac OS X or Windows, or any number of smartphones and other devices. Agents launch applications and test scripts (automatic or manual) and communicate test results. The developer sees
a local representation of the device under test on his Windows workstation. “When you press a button, you are really pressing the button on the physical [device] thousands of miles away,” Hahn said, noting that test results are displayed instantly on the developer’s screen in software. Pricing starts at around US$38,000 for a five-developer deployment. Agents also deliver test results to a Web-based test management client and a SQL Server–based repository. The repository also stores and facilitates reuse of
Testing Without Coding: GUIdancer Makes the Leap GUIdancer is an Eclipse-based environment from Bredex for testing Java/Swing application GUIs that the company claims requires no coding. New in version 1.1 is the ability to execute batch tests from a command line, support for testing multilingual applications using the same test components, new capabilities for trees and tables, and an improved user interface. Released in October, GUIdancer 1.1 (www.guidancer.com) for Linux, Mac OS X, Solaris and Windows also now includes an API to extend the tool—without code—for applications that use nonJava/Swing components and actions. In a statement announcing the release, Bredex managing director Hans-J. Brede said, “GUIdancer has a large repertoire NOVEMBER 2006
of components and actions, but there are often variations in their behavior in applications,” referring to the extensible GUI as the most important feature in the release. “We believe that testers shouldn’t have to program to create tests, even for modified components.” To avoid the need for programming, GUIdancer employs a so-called specification concept, which creates modular, reusable tests from source code before an application is completed. On the desktop, the program requires Java 1.5, which is included with the Windows download. The server component runs on Java 1.3. Demo and fully licensed versions can be downloaded now at http://www.bredex.de/en /downloads/downloads.html.
Device makers and telecom carriers take note: TestQuest’s new .NET-based platform can launch and control tests remotely.
test scripts, test requirements, information about the devices and user dashboards. A management client handles test assets, delivers them to devices and schedules tests. Test scripts are created with a standalone drag-and-drop environment written in .NET. The entire platform is extensible to enable customization and integration with an organization’s existing test equipment and processes, and to work with operating systems not included with the package. CountDown includes agents for mobile devices from LG, Motorola, Nokia and Samsung running Linux (and other open-source operating systems), Symbian OS (S60 and UIQ), Windows and Windows Mobile. Out of the box, the solution is most valuable to the development teams of device makers and telecom carriers. For the enterprise, its value is its ability to provide a testing solution for homegrown devices and proprietary operating systems or applications. “They still have a requirement for functional testing,” said John Yuzdepski, TestQuest chief marketing officer, of such developers. “Their devices are not cell phones—they are specialized devices from Symbol and others—but they have all the issues,” and often lack test tools for their self-made software. “So our ability to do end-to-end testing makes this an [ideal] tool.” www.stpmag.com •
9
Seapine Shores Up TestTrack TCM Seapine Software has added test case management to its line of application life-cycle and business process management solutions. TestTrack TCM, set for general availability on Nov. 1, will allow QA teams to collaboratively plan, build and manage creation of test cases and handle test scheduling and execution, as well as measure, analyze and report the results, the company claims. “Thorough test planning and test case creation are precursors to a successful quality assurance effort,” said Richard Riccetti, Seapine president and CEO in a news release, in which he described the new product as a solution for planning the testing process that can “manage thousands of test cases and the results of their execution.” TestTrack TCM builds on the company’s TestTrack issue-tracking architecture for Linux, Mac OS X, Solaris and Windows. TestTrack is a cross-platform framework for monitoring defects, bugs and feature requests and facilitates collaboration between all team members, including engineers, testers, developers, managers and technical writers. Rules can be configured to automatically assign issues to appropriate staff. Seapine is set to begin shipping Surround SCM 5, an update to its software configuration management platform that Riccetti claimed offers more than the simple “checked-in, checkedout” file states of competitive SCM tools. “Source code may undergo a code review and either pass or fail,” said Riccetti. Having the file move forward to the build process, he added, may hinge on the code reaching a certain state of quality. “With Surround SCM 5, it is easy to define states and policies to enforce best practices” or facilitate other departments that might require a structured change process. “Now if a developer checks out a ‘code reviewed’ file and makes a change, Surround can automatically reset its state to ‘not code reviewed’ to accurately describe the state of the file,”
Riccetti said. Metadata capabilities also include ownership tracking of all files. Also new in version 5 is hyperlink access to files, CVS-style check-outs, single sign-on support and tighter inte-
gration with TestTrack Pro. “Developers [now have] access to changelists from within TestTrack Pro and the ability to view all defect data from within Surround SCM,” said Riccetti.
A Pragmatic Approach To Project Management Pragmatic Software has added project management capabilities to Software Planner, the company’s Web-based software development life-cycle tool for requirements, test/QA and issue tracking. Delivered in a new Professional Edition for US$65 per concurrent user per month, management features include project plans that can be built and tracked within existing projects, with the ability to create tasks and assign them to individuals. Dependencies also can be created for tasks, thereby preventing one from beginning until another is completed. Members can see only tasks that apply to them. Managers can create time sheets, set baseline dates and costs, and follow variations as the project moves along. Multiple Software Planner now offers PM capabilities.
projects can be viewed within a single screen, giving project managers a simple status overview, the company says. The new version also synchronizes with Microsoft Outlook. Software Planner updates team members through e-mail when new tasks are assigned to them, who in turn input hours and percentage figures via a Web-based interface. Managers can then perform timesheet analysis for client billing. Clients also can be given a view of project status. Pragmatic also offers a $35-per-month Standard Edition without project management capabilities and an Enterprise Edition that includes project management and integrates with Crystal Reports and Data Dynamics’ DynamiCube pivot table reporting tool. This version has a one-time cost of $1,600 per concurrent user.
Send product announcements to stpnews@bzmedia.com
10
• Software Test & Performance
NOVEMBER 2006
Pea k Perfor mance
After All This Time, A Tool Maker Listens! loved the tool’s features, For over six years now, I’ve pricing model, paradigm been doing my best to proand extensibility, but vervide meaningful feedback sion 1 was still somewhat to testing tool vendors— narrow in its overall capaparticularly performance bilities, and a stack of those testing and load generation sticky-note requirements tool vendors. I’ve provided were yet to be implementfeedback as a paying client, ed. Nevertheless, against my as a services partner, as the better judgment, I started author of over 30 articles having hope—hope that about a particular tool, as Scott Barber this testing tool vendor a target of market would build a tool that was actually researchers who call when doing their designed, implemented and marketed quarterly stock valuations, and as someto help good testers test even better. one with enough voice in the field that I was a little surprised a couple of vendors often ask me for endorsements months ago when I received an invitaof their products. For over six years now, tion to attend an SDR (Software Design my feedback has apparently been disReview) for future releases of this suite missed or ignored—until now. of tools, but I’d been to these shindigs About two years ago, a vendor startbefore. The vendor gets a bunch of ed asking questions—the right quesexecutives from its big clients together tions. And not just of me, but of a relain a room for a day immediately pretively large group of software testing conceding its annual users’ conference. The sultants, authors and test tool users. I product managers talk about how great was pleased with the questions that were the next release is going to be; so great, being asked, but I was understandably in fact, that “you executives won’t mind hesitant to get my hopes up. I was more paying a little more for all these new feapleased when I was hired to do a feature tures, right?” Then the “technical sales and market analysis for the performrepresentative of the year” starts ance-testing aspect of a tool that was “demonstrating” the next version of the being built from scratch. And I was product by clicking through and recitdownright shocked when I arrived oning an obviously scripted set of words site to find notebooks full of printed and activities intended to impress the materials with such labels such as Open Forums, Vendor Forums, Roland Stens and roomful of executives who’ve probably Scott Barber, all with sticky notes hangnever even seen the previous version in ing out, each featuring a requirement action. All the while, the product mannumber. These guys were really doing ager is virtually chanting, “See, we heard their homework. your feedback and we added this new feature... See, we heard your feedback Hope Springs Eternal and we fixed that rough spot...” Finally, I got a chance to evaluate the beta and the executives are invited to fill out their then to use the version 1 release, and feedback forms with wish lists for next again I was pleased—but not thrilled. I year—which they dutifully do, with at
12
• Software Test & Performance
least some of the items on the list that their test managers gave them before the SDR. Then someone announces that the bar in the reception hall is open, at which point the executives dutifully submit their half-completed wish lists of items they barely understand anyway and go fetch their free beer. Same old, same old—right?
Not Just Business as Usual Wrong. Let me tell you, this was different—and it was different in a lot of surprising and positive ways. This wasn’t a one-day meeting in conjunction with a conference; it was three full days in conjunction with absolutely nothing else. And it wasn’t a one-time event tied exclusively to the next release of the product; it was one of a semi-annual series of SDRs focused on everything from hot fixes to the five-year product vision. Not only that, but the vendor covered the hotel expenses for the attendees. On the first morning when I took a seat and looked around the room, I noticed that instead of executives and sales partners, I was surrounded by the tool’s actual users. How did I suss this out, you ask? Two ways. First, I actually knew several people in the room from other venues. Second, the people I didn't know weren’t wearing ties, carrying day planners or exchanging business cards—they were wearing whatever was clean when they packed, were busy on their laptops taking care of whatever emergency had come up overnight, and were asking the people next to them if they knew a better way to solve the problem. Yes, these were genuine consultants and clients from around the world with actual firsthand experience using this tool on real projects—and many had also used the tool’s competitors. To tell the truth, I don’t think they were all fans of version 1, either. What I know for sure is that every single person in the room had his or her very own personal wish list for version 2 that he or she felt very Scott Barber is the CTO at PerfTestPlus. His specialty is context-driven performance testing and analysis for distributed multiuser systems. Contact him at sbarber @perftestplus.com. NOVEMBER 2006
strongly about and wanted to share with the group. As the meeting kicked off, I realized too that this event wasn’t run by a product manager and a technical sales representative, but an entire staff. In fact, over the course of the three days, 30 members of the company’s development, product management and marketing teams gave specific presentations, fielded challenging questions with honesty and grace, and solicited positive and negative feedback like professional facilitators. All the while, another 200-plus members of the company team were listening in via conference call. And just in case anyone who should have been there couldn’t or didn’t make it, audio and video were recorded throughout the entire three days. No, they didn’t tape the demos. (Yes, we did get some demos, but I had to sign a comprehensive NDA, so I’m probably not even supposed to tell you that one of the new features they previewed was so cool that it actually left me speechless... much to the amusement of the others in the room. Luckily, I got the vendor to review this column and approve it as “suitable for public consumption.”) What they did tape was all of the attendees providing feedback, opinions, praise, rants, expectations and wishes. And not only did they record it, they had some mechanism (that I’m going to have to get them to explain to me someday) to index the recording by keywords so folks could actually do searches on the recording later.
Somebody’s Listening As if all that wasn’t enough, it was clear throughout that the entire event was designed to solicit feedback—not to impress us with the next generation of “cool, but otherwise fairly useless” new features. In fact, the facilitators went so far as to list all of the potential enhancements under consideration on flip charts and have everyone in the room vote for a top 10 and bottom 10 to help them determine feature priority for the next release. Interestingly, by the end of the third day, it was obvious that on some features, the attendees had a very different recommendation than the vendor expected, and the vendor seemed NOVEMBER 2006
to take this message to heart. Of course, this all could have been a show to make us feel validated and listened to so that we’d all go start singing the praises of this vendor and the new version of its tool—but I don’t think so. Why? First, most of the SDR attendees are now part of a volunteer advisory board that has a virtual meeting twice a month and receives pre-beta software to test and provide feedback for, administrated by many of the same 30 individuals who ran the SDR. Second, many attendees are being recruited to develop training materials for the tool suite, compose contextually relevant help to ship with the product, and even write books in support of the next release. My third reason is personal: I know, respect and trust the man driving the vision of this tool. I don’t always agree with everything this gentleman says, writes or envisions, but I do believe that he is a brilliant man driven to do whatever he can to enable everyone involved in creating software to create better software. So do you want to know who this guy is? I suspect that many of you are already familiar with Sam Guckenheimer’s contributions to RUP (the Rational Unified Process). Now that Sam works for Microsoft, the company has presented the professional opportunity of a lifetime, inviting him to use his 25 years in software development to help teams develop software better, and to do it the way it should have been done in the first place. Further, Microsoft offered to build a supporting tool set from scratch based on Sam’s plans. (To find out more about Sam’s vision for software development, read his book “Software Engineering with Microsoft Visual Studio Team System” [Addison Wesley, 2006].) I’m sure you already figured this out by now: The vendor I’ve been crowing
•
about is Microsoft, and the version 1 tool is Visual Studio Team Suite 2005— which includes tools designed to assist and enable every member of a software development team to do his or her job more effectively and efficiently. I felt compelled to share this with you not simply because I'm a fan of the method Microsoft’s using to collect feedback about their tool. What had the biggest effect on me is the end result of the method as compared to what I have witnessed from some other vendors in the past. Over time, Microsoft’s feedback method will end up helping to develop a tool that users like and want to use. In contrast, with the methods similar to the one I described at the beginning of this column, the vendor ends up developing a tool that’s easy to sell to executives. Naturally, these vendors see “easy to sell” as a sign of success. But the inherent problem with “easy to sell” as a measure of success is that it doesn't take into account the percentage of the time that the tool becomes either shelf-ware or the beté noire of the folks who are forced to use it. If Microsoft stays the course with this method of collecting and applying feedback, going ahead to develop high-quality software addressing that feedback, this tool won’t have time to sit on the shelf due to its huge following of happy and loyal users. Certainly, none of this is a given, but it makes Visual Studio Team Suite well worth keeping an eye on. If the version 2 tool has an official name, designation or target release date, I don’t know about it. I do know the tool’s code name, but I’m not allowed to tell you. So you’ll just have to wait until the beta release, when I’ll give you a full feature review—and maybe then Microsoft will let me divulge the code name! ý
Over time, this feedback method will end up helping to develop a tool that users like and want to use.
•
www.stpmag.com •
13
Automation SDLC Avoids Teams Running in Circles
How to Build Tests and Apps In Tandem By Bob Galen
We’ve invested heavily in automation. We’ve purchased the tools and the training. We’ve focused one tester fulltime on automation development and allocated 20% of everyone else’s time for it as well. Three months into things, and we’re struggling to gain any automation traction. I don’t see any improvement in our overall testing cycle times. In fact, testing seems to be taking longer. What’s wrong? Do I have the wrong tool, wrong team, wrong approach, what? I need automation to be working… now! My initial response to their concerns is to remind them that automation is a software development effort. In fact, that’s exactly what it is. I explain that it takes time to start most worthwhile automation projects and that many organizations struggle in the beginning at properly setting the stage. Another important aspect of the challenge is that all automation efforts have to integrate with a development project—so resource and schedules are often shared. Instead of having a standalone project, you have two interrelated NOVEMBER 2006
development projects running in parallel, with automation understandably taking the priority back seat to its development counterpart. While some test teams are getting better at it, I still see far too many clients that manage their automation outside of good software development practices. I’d like to see that trend slowed or reversed. In this first of three articles on critical automation start-up management activities, I’ll explore four key ways to keep your automation efforts in step with traditional software development life-cycle activities: • Establish the major drivers for creating an Automation SDLC • Explore a few of the key success criteria behind a solid Automation SDLC effort Bob Galen is the author of “Software Endgames” (Dorset House, 2004). The founder of RGCG, a technical consulting company focusing on agility and pragmatism, he has more than 25 years of experience as a software developer, tester, project manager and leader. He can be reached at bob@rgalen.com.
Photograph by Darryl Sleath
F
requently I engage clients in assessing their automation efforts.The discussion normally begins with their expressions of frustration:
www.stpmag.com •
15
AUTOMATION SDLC
• Review Automation SDLC extensions from your own product SDLC • Consider how to correctly integrate automation with your mainline development efforts
Drivers for an Automation SDLC There are three key needs driving an Automation SDLC. First, it helps to align your automation development methods with your software development methods. In this way, the various teams, managers, project managers and most impor-
quality within the automation effort. Again, I often see test developers who simply dive in and begin automating test cases without a thought toward architecture or strategy—and we need to stop doing that. Third, an Automation SDLC also serves as a vehicle to synchronize with the SDLC. Since the phases and efforts are obviously tightly coupled, it helps to have things well defined so that interdependencies and integration points between the two life cycles are more easily understood. While this is
TABLE 1: ARCHITECTURE BUILDING BLOCKS Team Elements
• Charter your team with clear architectural and leadership roles • Assess and establish a base level of programming/ development skills • Develop training materials, online guidance, templates and checklists • Establish mentoring or pair-testing relationships • Obtain training for writing, running and framework automation development and maintenance activities
Process Elements
• Obtain or develop coding conventions, templates and design patterns • Adhere to naming conventions • Use proper test-case size guidance for decomposition, granularity and estimation • Establish review, promotion and removal models for test cases within the regression framework
Tool Elements
• Map tool selection criteria to your applied automated testing model • Use conventions for configuration management and defecttracking tools • Define wrapping conventions that target how you will insulate dependencies and workarounds
Architectural Elements
• Decompose AUT into meaningful layers or components for test case development and efficient regression testing • Establish a versioning model for controlling “packages” of tests for new feature and maintenance updates • Wherever possible, define libraries of reusable, well-tested automation components • Develop standards for logging errors (files, formats, review practices) and interfaces to reporting systems (logging, Web reporting, defects) • Gather lab or H/W interface information for test environment interrogation, scheduling and control • Use traditional test management approaches and/or tools for scheduling, results interrogation, defect reporting and metrics
tantly, senior executives can better understand your automation efforts. Essentially, it will level the playing field for understanding automation’s goals and challenges—and how it fits within the overall product development life cycle. The second driver is establishing a framework that illustrates the processes, phasing and release cycles associated with proper automation development. This enables improved planning and
16
• Software Test & Performance
more of a planning-level point, it also helps with tactical decisions; for example, determining feature readiness for subsequent automation development.
Key Success Criteria There are several considerations when defining and connecting your automation efforts to an SDLC. While this section isn’t intended to be exhaustive, it illustrates automation
concerns that are similar to those encountered by software developers building great products. Ignore these key success factors at your own risk. Most of these aren’t a day-one problem, and you can often skip many of them in the short term and gain a sense of progress and contribution. However, over the long term, a successful Automation SDLC must establish sufficient context for these foundational points. Business Case – The first thing to establish is the high-level business case for your automation. Part of this effort is establishing clear goals that link to the stakeholder and business expectations. In a follow-up article, I’ll spend more time on the components of a solid business case. Here I’ll focus on goals, from the cost, time and scope perspectives: how much automation coverage (scope) is required, within what timeframe and supporting what project delivery targets (time), and limited by what budgetary (cost) factors. Another way of looking at the business case is developing a charter for your automation project. The primary purpose of the charter or business case is to ensure everyone understands what the automation effort will actually do for the project or organization— and, equally important, what it will not do. This is the first step to combating the confusion I often see in key stakeholders who fundamentally don’t understand the value proposition, challenges and, most important, reccurring costs associated with a solid automation program. Architecture – A second important consideration is architecture. While today’s testing teams seem to be gaining in the ability to define proper automation framework architectures, architecture remains one of the tougher challenges. Don’t create automated scripts without defining an architecture that will support their development efforts in the long term. Even if you purchase off-the-shelf tools, you still need to wrap them with a thoughtful model to support your efforts. Beyond architecture, you also need to agree on the appropriate automated testing model for your product domain, skill set and requirements. Some of the leading approaches can be sorted into the following three categories: • Tool-driven record and playback NOVEMBER 2006
AUTOMATION SDLC
• Data, keyword or action driven • Structured, modular or programmatic engine–based Moving from simple to complex, those approaches increase in difficulty of initial development effort as they decrease in effort for ongoing maintenance. To build infrastructure, the more complex models also require development-savvy test engineers. Table 1 defines the elements that I consider essential for a properly defined architecture. Consider it a thinking tool rather than an exhaustive checklist. Requirements – Among your greatest challenges as a tester are the requirements gathering and management processes. You’ve been there: Requirements are late in development, slow to solidify and quick to change—often very late in the game. All the while code is quickly being written. It’s one of the ongoing battles in most software projects. Therefore, you have relevant experience with the need for requirements and the problems associated with their absence. Don’t fall into the same trap when developing your test automation architectural framework and automated test cases. Take the time to develop a solid set of requirements. Ensure that you include key development and business stakeholders in this process and construct the automation capabilities vision collaboratively. Who knows, your example may even become a model for your development team partners to copy. Implementation – Rarely is it a good idea to start with test case #1 and automate through test case #10,000. Instead, you should come up with an automation implementation strategy that guides you toward your goals as established in your business case and requirements. Often the early strategy is simply learning and establishing your automation framework. At this stage, rarely is there much focus on return on investment. Instead, you’re simply experimenting to find the right automation tools and techniques that will work within your product domain and project culture. However, you need to quickly redirect this strategy toward more traditional automation goals, such as increasing coverage, finding defects faster and improving overall quality of NOVEMBER 2006
the production product. In next month’s article, I’ll explore test case selection criteria in more detail. Project Synchronization – This final success criterion extends from your product development SDLC. Clearly, your automation efforts have to be effectively extended from the base product SDLC. While you can and should extend this into your Automation SDLC, which is the topic of the next section, you really need to understand the nuances—both technically and from a business and project management perspective—of each project you’re attempting to automate.
Extend Your Product SDLC Surely you don’t want to reinvent your own life cycle for automation unless you absolutely have to; automation is inextricably linked to your development methodology. For example, if you’re using one of the agile methodologies for product development, say Extreme Programming, your automation focus will be heavily influenced by its short iterations and acceptance testing demands. Another reason is simply to avoid confusion. Having separately defined methods will often require ongoing explanations across your project teams. It will also make you define some sort of integration view between the two methods. And it will drive your project managers crazy as they try to schedule across the two different perspectives. Instead, I prefer to extend the existing SDLC methods to include considerations for automation development. This isn’t too onerous because they typically already include testing activity anyway. In this section, I’ll discuss a few of the more critical parts of the extension model for automation—thus creating a group understanding of the Automation SDLC.
Extension #1: Make Automation Decisions From Completed Manual Test Cases Don’t change your manual test-case development process for automation. In fact, you might even want to tighten it up a bit. Only when the manual cases have been developed, executed and proven to be correct should you analyze the need to automate and
PREVENT R EWORK WITH SIMPLE COMMUNICATION I was working in an organization that made a tremendous investment in automation—committing to achieve nearly 95% automated test-case coverage, from one release to the next. To meet this objective, we established a parallel test development team to include new automation in the current product release schedule. In my experience, this is a rare automation strategy because of the substantial costs of running two testing efforts in parallel. However, simply having the people automating features evolving within a product’s release cycle wasn’t the only challenge. In this environment, we were developing UI functional automation for a storage platform–management application that was also evolving. Since the application was new, the interface was highly volatile as we exposed it to customers and gleaned their input on look and feel, feature operation and usability. This volatility impacted our automation efforts to varying degrees. In many cases, we’d be blindsided by a “simple” UI change that could cause us massive rework within our test automation. Because we were running in parallel, this had a direct impact on our testing project and release criteria. We finally decided to sit down with the development team and sensitize everyone to our automation architecture and approaches, trying to share the impact— both trivial and major—that various development changes could have on our automation. This impact- and pain-sharing created a good deal more understanding between the two teams. While we still received many changes that impacted us, the number of changes with severe impact dropped to near zero. Cross-team communication improved also. This is one of the valuable side effects of a solid automation development effort—bringing the two teams closer, gaining technical impact understanding, and improving collaboration. And in this case, it drove mutual respect as well.
schedule them for automation development. You should take this approach for several reasons. First, it helps you to have a consistent format and workflow for developing all of your test cases. It ensures that the test cases are essentially “complete” before you start worrying about automation. It also allows you to queue up a set of well-defined work items that can be rearranged as priorities and needs change. www.stpmag.com •
17
AUTOMATION SDLC
Think of manual test-case development as the early design phase for your automated test cases. The process will also sort out many of your requirements clarity issues before you ever start coding. Finally, if parts of features are not functional or are problematic, you’ll be aware of them in advance of automation development.
The reviews are a wonderful place to share ideas, techniques and tools around testing each feature. The key to good automation requirements definition is to begin blurring the lines a bit between the development and testing perspectives.
Extension #2: Review From an Automation Perspective
If I’ve learned one thing from operating in agile teams, it’s that short iterations work extremely well in providing feedback on your direction in software projects. And what’s true for software development is even more applicable when developing test automation. When developing any automation effort, you’re stuck behind two requirement curves: the product and the automation. You’re also trying to keep up with the implementation evolution for the product. These three forces create a lot of uncertainty and drive up risk. The only way to combat those risks is with a finely grained iterative model.
Extension #3: Establish a Highly Iterative Automation Model
I’m assuming two review points in your traditional process: first, that testers are invited to requirements review sessions; and second, that testers expose their test cases for external review as part of their test development process. Even though you’re deferring automation development decisions, you do want to surface automation concerns throughout the artifact review process. Often, automation requires data and environmental control that may drive product enhance-
FIG. 1: THREE-PHASE AUTOMATION MODEL
Application release v1.0 – Construction
Application release v1.1 – Construction
v1.0 Feature Testing & Regression
v1.1 Feature Testing & Regression
v1.0 Automation Development
ments to truly enhance the effectiveness of the automation. You want to get these on the table early on. You’re also looking for prioritization information on testing attributes for speed of execution and cycle time and/or coverage, so that you can focus your automation efforts on higher impact areas. The overall product technical risk areas should also come into play—driving automation decision-making. When reviewing requirements, ask questions about feature dependencies and testability. As you begin to automate more heavily, you’ll notice more synergy between development unit testing and your automation efforts.
18
• Software Test & Performance
Application release v2.0 – Construction
v2.0 Feature Testing & Regression
v1.1 Automation Development
One of the challenges facing that model is integrating your automation with the product release stream. Rarely do feature delivery and functionality align perfectly to support your automation efforts. I’ve become quite creative at negotiating early, finely grained functional releases from development whose primary purpose is to help my automation efforts. Call them automation alpha candidates or something similar—they foster a willingness to share early and often. This has a twofold benefit. It allows the automation to be developed in small pieces iteratively, while uncovering product defects early. The downside? You’re implementing automated tests on unstable features that may drive higher
degrees of rework, but I think that’s an acceptable trade-off.
Extension #4: Hand-offs for Infrastructure, Automation and Execution From a workflow perspective, it’s particularly important to have automation handoffs defined properly. By hand-offs, I mean the internal release of automation packages within your teams. Think about how you want software delivered to you for testing—with release notes, exit/entry criteria, and in some sort of versioned package that has been tested. These same sorts of workflow process exchanges need to be planned and executed for automation deployment as well. If you’ve opted in your organization for a three-phase automation model as illustrated in Figure 1, these hand-offs should align across those boundaries: • Infrastructural components are created or modified, packaged and released to the automation development team (or individuals) for test development. • Completed, tested and packaged automated test cases are released to the automation execution team (or individuals) for execution. • The regression or automation execution team (or individuals) run the automation and provide feedback to the test developers for enhancements, efficiency/speed improvements or general bug fixes. Following this workflow not only helps to manage expectations and planning for project support, but defines roles and expectations for deliverable quality within the automation development team.
Extension #5: Extend Traditional Tools and Processes for Automation Support Don’t forget that automation requires many of the same tools and practices associated with production software development, including the configuration management/version control, bug reporting and change-control tool sets and processes. Rarely is it a good idea to create your own environments for these tools. Instead, you’ll want to extend your existing tools to include automation development as a project or series of projects. And of course you’ll need naming and version conventions for your automation development. You’ll also want to get good at prioritizing defects (in automated tests) with cusNOVEMBER 2006
AUTOMATION SDLC
tomer input (from the test runners). Planning and packaging is important too. On larger automation efforts, I’ve even instantiated more formal change control and a change control board, just to bring some rigor and control to the automation development process. The more reuse of tools, techniques and process you can create here, the more efficient your automation efforts will become, simply because you don’t have to train your team or establish unique tools and processes. You can even use examples to lead the product organization by exposing best practices from the automation team and suggesting they be adopted more broadly.
Extension #6: Analyze, Increase and Communicate Runtime Efficiency As your automation development matures and the overall coverage increases, it becomes more and more important to the project management delivery perspective. Thus, expect the PMs to take more and more interest in your plans, schedules and overall execution performance. You’ll want to take the time to explain the nuances of your automation strategy to them and to other stakeholders. Discuss where the strategy can increase overall testing efficiency and where it can’t—then work with them to tailor your plans to maximize project impact and success. Make sure to share some of the challenges associated with running automation, including automation maintenance, debugging and fault isolation, test environment management and tools integration. Draw on the similarities to traditional software development as a means of effectively communicating automation state. You’ll also want to share your efforts at making automation runtime more efficient, and how it can actually become a speed differentiator in supporting your development projects. This becomes an imperative if you’re doing any outsourcing or implementing distributed automation development and/or execution. While stakeholder communication is important at the beginning of your automation efforts, I consider it an ongoing priority and an extremely important part of your role. You’d be surprised how many stakeholders seem to forget about the value proposition and nuances of test automation, and NOVEMBER 2006
FIG. 2: HANDLING COMPRESSION AUTOMATION
Automation Infrastructure
Application Features
Initial Product Release
Automation Development
Iterative Product Releases
how often they need to be reminded.
Extension #7: Properly Connect Automation to Your Project Phasing Any automation effort is deployed within the context of a higher-priority project—the development project it’s associated with. This brings into play an interesting set of dynamics in which the two are inextricably linked, but one clearly outshines the other. Typically, what happens—and it should—is that the automation delivery is skewed or phased outside of the mainline product development activity. The testing team becomes responsible for testing the product (the prime directive) while trying to develop automation on early features that are still stabilizing and maturing. The degree to which you can develop this automation and have it deployed and usable within the current product iteration is dependent upon several factors: • The organization’s product SDLC • How quickly features within the software are maturing • Whether the software development and testing teams are on schedule • Overall resources available to the SQA team Figure 2 illustrates the typical skewed development cycles for automation within most development projects. Essentially, automation is delivered one release behind the product delivery schedule. This allows the features to stabilize and offers sufficient time for proper automation development and deployment into your regression testing cycles. It also keeps automation development off the critical path of the primary development project. While you can achieve compression
Package Hand-offs
Automation Execution
Final Release
of automation into the current release, it requires many of the factors listed above to be mitigated. It also is far more resource-intensive if you skew the automation work to be truly parallel with product development. Figure 2 offers a good example of how to handle these compression challenges.
Automation Can Lead the Way The key point is not to reinvent the wheel. Instead, leverage your existing product SDLC and extend its tactics, processes, tools and practices to include automation development. If your organizational practices don’t include solid development practices—for example, chartering, architecture development, requirements definition and solid reviews—simply lead the way by implementing them from within your test automation development efforts. Also, never lose sight that automation development is a software development activity, and all of the field’s rules and methods should clearly apply. Make sure that your stakeholders understand this. Finally (although this topic isn’t formally explored in this article), don’t allow your automation efforts to become a critical path item within your primary product development schedule. Instead, skew them out so that you have the time and focus to implement them correctly and without derailing the project. Establishing a connected and clearly defined Automation SDLC is one of the first steps toward your automation path. In the next article in this series, we’ll discuss something equally challenging—how to pick the right tests to automate. ý www.stpmag.com •
19
By Anna Mikhailova
A
public Java API can be designed by one developer, implemented by another, and extended by a third. The resulting application can then be delivered to end users. To ensure high initial quality and to make certain that subsequent implementation changes won’t break user extensions, a public API must be designed and implemented with a proper use of application domain modeling techniques and architectural design patterns that support maintainability and robustness. Last but not least, an API must be designed and implemented in such a way as to enable extensive and thorough testing. If due consideration isn’t given to quality assurance at the design phase, testing can prove to be ineffective after the API is designed and implemented. Industry experts recommend that as
20
• Software Test & Performance
much as 80 percent of code be written entirely in terms of interfaces, not concrete base classes.1 This approach has a number of advantages from both a methodological and a practical point of view. Using interfaces as specifications of behavior and building class hierarchies based on interface inheritance instead of implementation inheritance helps to provide a clean separation between semantic inheritance relationships and code reuse. From a practical standpoint, substituting an implementation hidden behind an interface is much easier and less errorprone than replacing a concrete base class that might have been reused. In its online guide to writing doc comments for the Javadoc tool,2 Sun establishes the following guidelines for a Java Platform API Specification: • The Java Platform API Specification is defined by the documentation comments in the source code and any documents marked as specifications reachable from those comments. • The Java Platform API Specification is a contract between callers and implementations. The Specification describes all aspects of the behavior of each method on which a caller can rely. It does not describe implementation details, such as whether the method is native or synchronized. NOVEMBER 2006
• Unless otherwise noted, the Java API Specification assertions need to be implementation-independent. Exceptions must be set apart and prominently marked as such. • The Java API Specification should contain assertions sufficient to enable Software Quality Assurance to write complete Java Compatibility Kit (JCK) tests. Essentially, these guidelines are established to make sure that an API specification is declarative rather than operational, precisely stating what a method does but not how it does it. Somewhat contrary to an intuitive belief, a specification can be both precise and nondeterministic. For example, for a square root–calculating function that takes a float n as input and returns a float r as a result, a specification can state that r ^2 = n. This doesn’t state how exactly r is calculated, but it tells you about the effect of the operation. Also, these guidelines stipulate that specification and implementation synchronization should be easily (and ideally automatically) verifiable. Otherwise, inconsistencies between specification and implementation will lead to maintenance problems. These guidelines apply equally well to Anna Mikhailova is a principal consultant at Software LifeCycle Group. Previously, she was a senior consultant with Parasoft. She has written more than a dozen scientific and technical articles on software quality and verification, as well as object-oriented design and development. Contact her at amikhailova @adelphia.net. NOVEMBER 2006
specification of all public Java APIs, not only Java Platform APIs, and will prove equally valuable to the specification of all public Java APIs.
Challenges in Testing an API With an open Java API, traditional methods of functional testing are insufficient, because there is no application that one can test against use case scenarios. There can even be no specific use case scenarios until the object-oriented framework is fully implemented and extended to become an application. Unlike functional testing that focuses primarily on verifying that the expected functionality is correctly implemented in selected cases modeled as usage scenarios, unit testing focuses on uncovering errors at a unit level for all methods and all classes considered in isolation. In many cases, an API can be tested only at a unit level. The JUnit framework has long become an industry standard for unit testing. It allows us to abstract away from external resources and dynamic conditions to focus on the behavior of one method at a time. It’s widely acknowledged, however, that the value of unit testing is mostly in robustness testing or reliability testing. Unit testing exposes runtime errors that result from not processing all possible method inputs correctly or not handling all possible situations properly. In practice, a test case supplies “negative” values that lie outside of the assumed input domain to the method
Design Public APIs Using Application Domain Modeling Techniques And Design
Patterns www.stpmag.com •
21
THE PUBLIC JAVA API
under test and checks whether the method terminates normally or throws an exception. Unit test–generating tools can automatically generate such “negative” inputs through introspection, analysis of the method’s source code and other methods. This kind of testing is most useful because it reveals implementation errors—the true purpose of testing. Testing the expected behavior of a particular method or a class at a unit level is much more involved. In its simplest form, a unit test verifies that the method under test terminates normally and doesn’t throw an uncaught exception. To verify anything beyond that in a unit test, a developer testing his implementation of an API asserts the properties that he believes hold after the method call. To understand what can be asserted, the developer must have a reference of what the expected behavior of the method under test really is. When an API is specified using textual descriptions of the intended functionality expressed in code comments, understanding the intended functionality amounts to guesswork. And if the guess is wrong, unit testing won’t find the problem because the developer will be testing that the method under test behaves as he thinks it should, not as the API designer intended it to be. The problem will only be revealed after integration, at runtime. Errors in an API can come from two sources: inconsistency, incompleteness and ambiguity in the API specification that the API designer can introduce, on one hand—and misunderstanding by the implementer of the API designer’s intentions on the other. These are two sides of the same coin, and the blame can’t be solely placed on either side. They both lack a mechanism for properly communicating the expectations and the obligations that one party can rely on and the other promises to deliver.
Using Formalized Specifications
•
To be effective, testing in general and unit testing in particular should be carried out in a systematic and rigorous manner. When an API is defined using formalized specifications rather than textual descriptions of the intended behavior, we have a statement of how a method expects the client code to call it, so that this method can properly handle client inputs. When verifying the correctness of the implementing method, we therefore need to test the behavior of the method against input values that meet the expectation. Similarly, if we have a description of what a method is obligated to do, we can design test cases to check if it indeed does what it should. Formalized specifications can help the API designer and the API implementer to understand each other and speak a common language. The API designer can express his intentions precisely and unambiguously, while the API implementer can validate an implementation’s correctness with respect to this specification. This kind of correctness testing from formalized specifications results in systematic and rigorous verification of system behavior, avoiding the shortcomings of verification against text-based specification. What does it mean for an API implementation to correctly implement its specification—and how we can test the correctness of that implementation? Before exploring the answer, I’ll define the terms and discuss various relationships that bind actors in an object model and regulate communication and cooperation among these actors.
Formalized specifications can help the API designer and implementer speak a common language.
22
• Software Test & Performance
•
Semantic ‘Is-A’ Relation An object-oriented model is usually datacentric. Actors in an object-oriented model are implemented as interfaces or classes that group data and methods for processing this data, with an emphasis on aggregation/composition relations. Aggregation and composition are two variants of the “has a” relation.
Object-oriented modeling also emphasizes the so-called “is a” relation. On the implementation level, the “is a” relation is usually expressed in terms of interface inheritance. However, on the level of an object-oriented model, the “is a” relation signifies not only a syntactic relation, but a semantic one. The semantic “is a” relationship is sometimes referred to as a refinement relation and is defined as follows: A class A' refines class A whenever an instance of class A' is correctly substitutable in any context in which an instance of class A is used. When correctly used, the “is a” relation permits a designer to build a hierarchy of types. An instance of a lowerlevel type in such a hierarchy can always be safely used in any context declared to accept a higher-level type. The same definition of refinement holds for API specifications and their implementations. Refinement in this case stands for “correctly implements.” Whenever an implementation refines a specification—in other words, the implementation correctly implements the specification—client code can always use an implementation object in place of a specification instance. As an example, consider a class diagram for a geometry package shown in Figure 1. A geometry can consist of points, lines, curves, arcs, multipoints, polylines and polygons. An entire object model shown in Figure 1 is expressed in terms of interfaces, shown as white boxes. Concrete implementations of the interfaces are declared as final public classes, shown as gray boxes. If you need to plug in alternative implementations, you can always create custom concrete classes and use them instead of the default ones, as all interactions among the agents in the object model occur through interfaces. As shown in Figure 1, a public final class DefaultPoint implements the Point interface, and a public final class DefaultMultiPoint implements the MultiPoint interface. A client application can use DefaultPoint and DefaultMultiPoint directly. Or, if DefaultPoint’s implementation is unsatisfactory for some reason, the client application can provide a new implementation CustomPoint for the Point interface. Because MultiPoint communicates with Point only through its interface, MultiPoint will be able to use CustomPoints plugged in instead of NOVEMBER 2006
THE PUBLIC JAVA API
FIG. 1: AN OBJECT MODEL WITH CLASS
«interface» Geometry
1 1
1
*1 «interface» HighLevelGeometry
«interface» GeometryComposite *
«interface» Envelope
«interface» Segment
«interface» Path 1*
*1 «interface» Point
«interface» MultiPoint
«interface» PolyCurve<E extends Path>
«final class» DefaultEnvelope
«final class» DefaultPath 1
«final class» DefaultPoint
«final class» DefaultMultiPoint
«interface» Polygon
«interface» Polyline
«interface» Ring
«interface» Line
«final class» DefaultPolygon
«final class» DefaultPolyline
«final class» DefaultRing
«final class» DefaultLine
«interface» BezierCurve
«interface» EllipticArc
«final class» DefaultEllipticArc «final class» DefaultBezierCurve
0..1 «interface» CoordinateSystem
«interface» UnknownCoordinateSystem
«interface» ProjectedCoordinateSystem
«interface» GeographicCoordinateSystem
«final class» DefaultUnknownCoordinateSystem
«final class» DefaultProjectedCoordinateSystem
«final class» DefaultGeographicCoordinateSystem
DefaultPoints without any changes. As seen from this diagram, this design provides for using DefaultPoint wherever an instance of Point, HighLevelGeometry or Geometry is expected. To show that using a DefaultPoint instance in place of Point is safe and won’t break user extensions, you now need to verify that the class DefaultPoint correctly implements the Point interface. Before you consider how to achieve this, take a look at how you can specify the Geometry and Point interfaces. The Geometry interface is defined in Listing 1.
Specifying an API: Pre- and Postconditions To meet three criteria that a proper API specification should satisfy according to Sun—precision, non-ambiguity and nondeterminism—you can employ Design by Contract, a formal framework based on E.W. Dijkstra’s work on using preand postconditions for formal program specification and verification. Design by Contract is supported by various indusNOVEMBER 2006
try initiatives, most notably by its inclusion in Java 1.4, as described in Sun’s Programming With Assertions guide and other documentation.3,4 Three main types of assertions regulate the interaction between the client and the supplier code: preconditions, postconditions and class invariants. A precondition stipulates the supplier code’s expectations of the client code. A postcondition stipulates the guarantees that the supplier code promises to deliver to the client code if its precondition is met. A class invariant states the condition that is established by the class constructor and maintained by all of its methods, so that a class invariant holds after the constructor call, and also before and after each method call.
Preconditions: Restricting Input Domains Preconditions can be used to restrict input parameters to a subdomain of a type. The most common restriction is to accept only non-null inputs. For example, the method setCoordinateSystem () can be declared to accept only non-nulls:
/** * Associates the coordinate system with this geometry. * @pre coordSys!= null */ public void setCoordinateSystem(CoordinateSystem coordSys);
Using preconditions, you can express complex relationships on the pre-state of an object even if the method being specified has no input parameters. For example, consider the method snapToCoordinateSystem(). This method can be executed only on a geometry whose coordinate system already has a set precision. You can express this fact using the following precondition: /** * Rounds all coordinates to the precision defined by the geometry's * spatial reference system. * @pre getCoordinateSystem().hasXYPrecision() */ public void snapToCoordinateSystem() throws OutOfBoundsGeometryException;
Postconditions: Restricting Output Range Postconditions can be used to convey information about the outcome of a particular method. Postconditions can www.stpmag.com •
23
THE PUBLIC JAVA API
impose restrictions on the return parameters of a method as well as on the state of the target object. For example, you might want to state that the method getEnvelope() always returns a non-null object. Moreover, you can further specify that when the geometry is empty, the envelope object returned has zero width and zero height:
LISTING 1: GEOMETRY OF AN INTERFACE public interface Geometry { /** * @return the geometry's dimension */ public GeometryDimension getDimension(); /** * @return the unique Envelope that binds the Geometry object */ public Envelope getEnvelope();
/** * @return the unique Envelope that binds the Geometry object. * @post $result != null * @post isEmptyGeometry() $implies ((getEnvelope().getWidth() == 0) && (getEnvelope().getHeight() == 0)) */ public Envelope getEnvelope();
/** * @return the geometry's type */ public GeometryType getGeometryType();
/** * @return the coordinate system associated with this geometry */ public CoordinateSystem getCoordinateSystem();
You can also express a number of behavioral dependencies among the object model’s entities that the designer of the Geometry package can convey to the implementers using pre- and postconditions. For example, a geometry’s dimension depends on its type. A point and a multipoint make a zero-dimension geometry; a line, polyline, curves and arcs make a one-dimension geometry and so on. To express this dependency, you can use the following postcondition, as shown in Listing 2. This offers the most detailed and precise specification of not only the behavior of this particular method but also of a relational dependency between the classes GeometryType and GeometryDimension.
Pre- and Postconditions: Restrict Object Types As noted, in object-oriented modeling and design, the “has a” relationships are emphasized to a large extent. The inclusion of parameterized types, or generics, in Java 5.0 is very handy in this effort. By using parameterized types, you can make “has a” relations even more explicit, as the part type now contributes to the signature of the composite type. For example, in the class diagram shown in Figure 1, PolyCurve contains Paths. PolyCurve is homogeneous—it can contain only one type of Path; not Paths and Rings, for example. That’s exactly what the declaration PolyCurve<E extends Path> expresses: The type PolyCurve is parameterized throughout by the bound parameter E that must be a subtype of Path. Relevant articles can further your understanding of the benefits of parameterized types 5,6,7, but here I’ll just stress the bottom line: The proper use
26
• Software Test & Performance
/** * Associates the coordinate system with this geometry */ public void setCoordinateSystem(CoordinateSystem coordSys);
/** * @return whether this geometry is empty */ public boolean isEmptyGeometry(); /** * Sets Geometry to empty */ public void setEmptyGeometry();
/** * Rounds all coordinates to the precision defined by the geometry's * spatial reference system. */ public void snapToCoordinateSystem() throws OutOfBoundsGeometryException; }
of parameterized types helps in designing better and cleaner APIs, because parameterized types help to avoid type casting while providing all the benefits of compiler type checks. Preconditions can conveniently complement type structure and provide more insight into which specific types can be and are expected to be passed where a generic type is specified due to contravariance of input types. This is especially relevant when generic types are used. Similarly, postconditions can provide an insight into which specific types are returned from a method if this method returns an object of generic type. As an example, consider a method Polygon::constructUnion(), which is declared as follows: /** * Constructs the union of all polygons in the list passed
as the input parameter * */ public void constructUnion(List<? extends Polygon> polygons);
Recall that Polygon is an interface. Therefore, in the signature of constructUnion() it can only talk about Polygon objects thereof. To provide an implementation of constructUnion(), however, you need to work with concrete classes, such as the class DefaultPolygon that implements the Polygon interface. You can express this requirement using the following precondition: /** * Constructs the union of all polygons in the list passed as the input parameter * @pre polygons!= null * @pre polygons.$forall (Polygon p; p instanceof DefaultPolygon) */ public void constructUnion(List<? extends Polygon> polygons);
NOVEMBER 2006
THE PUBLIC JAVA API
LISTING 2: POSTCONDITION OF DEPENDENCY /** * Returns the dimension of the geometry object based on the geometry's type. * @return the geometry's dimension getGeometryType() == GeometryType.Null || * @post (( getGeometryType() == GeometryType.Point || getGeometryType() == GeometryType.Multipoint ) $implies ($result == GeometryDimension.Zero)) * @post (( getGeometryType() == GeometryType.Polyline || getGeometryType() == GeometryType.Line || getGeometryType() == GeometryType.CircularArc || getGeometryType() == GeometryType.BezierCurve || getGeometryType() == GeometryType.EllipticArc ) $implies ($result == GeometryDimension.One)) * @post (( getGeometryType() == GeometryType.Polygon || getGeometryType() == GeometryType.Envelope || getGeometryType() == GeometryType.MultiPatch || getGeometryType() == GeometryType.Ring || getGeometryType() == GeometryType.TriangleStrip || getGeometryType() == GeometryType.TriangleFan ) $implies ($result == GeometryDimension.Two)) getGeometryType() == GeometryType.Ray || * @post (( getGeometryType() == GeometryType.Sphere ) $implies ($result == GeometryDimension.Three)) */ public GeometryDimension getDimension();
Because preconditions can be weakened in a subtype (including an implementing final class), when providing an alternative implementation of the method constructUnion() that can handle a DefaultPolygon and a CustomPolygon, for example, you can have a weaker (more permissive) precondition for the constructUnion() method, as follows: /** * Constructs the union of all polygons in the list passed as the input parameter * @pre polygons!= null * @pre polygons.$forall (Polygon p; (p instanceof DefaultPolygon || p instanceof CustomPolygon)) */ public void constructUnion(List<? extends Polygon> polygons);
Similarly, postconditions can be strengthened in a subtype (subclass), which will allow for returning more concrete results from a subtype (subclass) object.
Correctness of Geometry APIs Specifications of interfaces that are higher up in the type hierarchy are inherited by the interfaces that extend them. In practice, this means that the Point interface specification is the same as the Geometry interface specification plus the specification of all the new methods introduced by Point. To show that DefaultPoint correctly implements the Point interface, you need to define test cases for DefaultPoint that respect the preconNOVEMBER 2006
ditions and postconditions specified in the Point interface and all its supertypes cumulatively. Consider a few examples. Recall that the method setCoordinateSystem() is defined as follows: /** * Associates the coordinate system with this geometry. * @pre coordSys!= null */ public void setCoordinateSystem(CoordinateSystem coordSys);
Automatic unit test generating tools, such as Parasoft’s Jtest, will respect the precondition and generate a test case that automatically creates a CoordinateSystem object and passes it as an input argument: public void testSetCoordinateSystem1() throws Throwable { CoordinateSystem coordsys = DefaultGeographicCoordinateSystem(); DefaultPoint THIS = new DefaultPoint (); THIS.setCoordinateSystem (coordsys); // No exception thrown }
If a postcondition is defined for a method, all test cases that are automatically generated can be evaluated for satisfying this postcondition. For example, the following test cases could be generated for getEnvelope(): public void testGetEnvelope1() throws Throwable { DefaultPoint THIS = new DefaultPoint (); Envelope envelope = THIS.getEnvelope ();
// No exception thrown }
Because getEnvelope is specified to have a postcondition isEmptyGeometry() @post ((getEnvelope().getWidth() == 0) && (getEnvelope().getHeight() == 0))
$implies
an automatic unit test–generating tool could automatically verify that this postcondition holds after the method under test terminates, by executing the corresponding checks and then verifying their outcomes. An appropriate exception can be thrown to inform the developer whenever this postcondition is violated.
Correct and Robust Automatic unit testing tools can go a long way toward verifying the correctness of Java APIs. To do this effectively and to address functional correctness in addition to robustness, you need a reference point, a precise specification of the behavior that the class under test needs to implement. Explicitly specifying the behavior in terms of preconditions, postconditions and class invariants provides such a reference point. Existing commercial tools can leverage the formalized specification defined for an API and automatically verify the conformance of an implementation to its specification. When an API is specified using preconditions, postconditions and class invariants, these specifications can also be leveraged to verify the correctness of system integration at runtime. And, by integrating such a tool into your development cycle, in addition to verifying the correctness of API implementations with respect to their specifications, you can also prevent systemlevel errors. ý REFERENCES 1. Holub, Allen. Why Extends Is Evil. JavaWorld, August 2003 (http://javaworld.com/javaworld /jw-08-2003/jw-0801-toolbox.html) 2. Sun Microsystems. How to Write Doc Comments for the Javadoc Tool (http://java.sun.com/j2se/javadoc /writingdoccomments/index.html) 3. Sun Microsystems. Programming With Assertions (http://java.sun.com/j2se/1.4.2/docs/guide/lang /assert.html) 4. Wm. Paul Rogers. J2SE 1.4 Premieres Java’s Assertion Capabilities, Part 2. JavaWorld, December 2001 (http://www.javaworld.com/javaworld/jw-12-2001/jw-1214-assert.html) 5. Generics tutorial (http://java.sun.com/j2se/1.5.0 /docs/guide/language/generics.html) 6. Langer, Angelika. Java Generics FAQ (http://angelikalanger.com) 7. Bracha, Gilad. Generics in the Java Programming Language (http://java.sun.com/j2se/1.5/pdf /generics-tutorial.pdf)
www.stpmag.com •
27
By Shaun Bradshaw
utomating test execution allows organizations to achieve improved test coverage, reduced test execution time, decreased
A
production errors and increased repeatability and reliability. After companies invest time, effort and resources to build a test automation environment, it’s important for them to capture costs and benefits in a meaningful manner to verify financial return on investment. Demonstrating financial success allows upper management to further justify investing in process improvement resources.
Calculating Costs vs. Benefits While the costs associated with test automation are relatively easy to track and to account for, the benefits can be tricky. Despite the claims of some test tool vendors, setup and implementation costs often go beyond tool licensing fees and additional hardware.
28
• Software Test & Performance
These added costs vary depending upon the test automation framework used. There are two sides of the equation for determining the financial success of a test automation implementation effort. When reviewing any process improvement program, the primary value derived from the effort can be calculated in terms of cost savings. To calculate the amount saved, you need an initial cost basis against which to compare. With that in mind, it’s important to establish metrics tracking prior to implementing test automation. Let’s look at each benefit related to test automation and discuss what metrics can be tracked to determine the comparison basis, as well as the value/savings derived from the automation effort.
How can you track test coverage, and what metrics should you use to indicate how much of the application has been tested? Start by assessing how many test cases can be executed during a manual test effort. Note that test coverage isn’t the same as determining how many defects exist or how many test cases have passed. It’s simply a measurement of how much application functionality was tested based on the number of test cases executed regardless of the outcome (pass or fail). Using the test coverage metric, simple analysis can determine an automation effort value. Use Tables 1 and 2 (see page 30) as an example. Across the four product releases, the test team in Table 1 averages 15 minutes per test, or about four tests per hour. Table 2 assumes that all regression tests are automated, that all tests are executed and that automated tests can be executed in two minutes each, or about 30 tests per hour. (The two-minute estimate is based on an ongoing automation business case study by David Dang of Questcon Technologies, Greensboro, NC, published 2003.) Comparing the results of the two tables, a 25 percent increase in test coverage can be achieved with no additional test execution time overhead. Increased test coverage allows an organization to test more in less time, utilize fewer resources to execute the same number of tests in the same amount of time, or both. Increased test coverage also improves the chance of finding defects that might otherwise make it into production.
Benefit 2: Reduced Test Time, Cost To calculate the savings associated with using automated test execution, it’s necessary to track two more data elements in addition to the information derived from the test coverage analysis above: the loaded costs of a manuShaun Bradshaw is director of quality solutions for Questcon Technologies, a software QA and testing firm.
NOVEMBER 2006
Photograph by Aaron Springer
Automate App Tests To Achieve Maximum Benefit
Benefit 1: Improved Test Coverage
al tester during test execution, and the loaded costs of an automated tester during test execution. Reviewing the test coverage analysis above, it takes less time to execute all of the test cases using automation than it took to manually execute approximately 95 percent of the new tests and 55 percent of the regression tests. In addition to increasing the test coverage capability of a test effort, the cost associated with testing is reduced, and time-to-market decreases. Applying costs to the execution time helps demonstrate the savings potential over the course of the four releases. Include the following two assumptions: Manual testers cost US$50 per hour inclusive of overhead costs, and automation testers cost $75 per hour inclusive of overhead costs. By applying these resource costs to the data elements used in the test coverage analysis, Table 3 compares a fully manual test effort versus a test effort that includes automation of the regression test suite. Based on this information, it can be deduced that as regression baselines increase in size (or in the frequency in which they need to be executed), the savings resulting from test automation also increases. This means that projects with frequent patches or new functionality will benefit the most from test automation. Continuing with the earlier example, over the course of the four releases, utilizing test automation can increase test coverage by 25 percent and decrease test execution costs by $2,304.
Tests Are Gauged By Code Coverage, Execution Time and Cost– Here’s How To Squeeze Every Penny From All Three
Benefit 3: Decreased Production Errors To predict the number of additional defects found using automated testing, you must understand typical defect rates when tests are executed for new
NOVEMBER 2006
www.stpmag.com •
29
REGRESSION TEST AUTOMATION
TABLE 1: MANUAL LABOR, BAD Project ID
Number of New Tests1
Number of Regression Tests
Number of Tests Executed1
Test Execution Time2
Required Test Execution Time3
Release v.1.0
200
0
190
46 hours
49 hours
Release v.1.0.1
20
150
103
25 hours
42 hours
Release v.1.4
80
165
172
42 hours
59 hours
Release v.2.0
160
235
290
71 hours
96 hours
1. Number of tests executed includes new functional tests and regression tests actually executed. 2. Test execution time includes time to execute the tests and to research and communicate defects. It doesn’t include delays occurring during the test effort that aren’t directly related to executing the tests. For example, if the DBA has to load reference data prior to test execution but is delayed because of a production problem, this type of delay shouldn’t be included. On the other hand, if executing a test case requires that a 2-hour batch job be executed, that delay should be included. 3. Required test execution time is the amount of time required to execute all available tests.
Note: It’s rare that every test case will be or can be automated. We make this assumption to simplify the example.
TABLE 2: AUTOMATION, GOOD Project ID
Number of New Tests
Number of Regression Tests
Number of Tests Executed
Test Execution Time
Release v.1.0
200
0
200
49 hours
Release v.1.0.1
20
150
170
10 hours
Release v.1.4
80
165
245
25 hours
Release v.2.0
160
235
395
47 hours
functionality, as well as the typical defect rate when doing regression testing. (The defect rates for new functionality and regression testing should be tracked separately, since regression testing will almost always be substantially lower than new functionality testing). My company has found that most development organizations can expect a defect rate of about 25 to 30 percent on new functionality, and 1 to 5 percent during regression testing. Regardless of whether you execute tests manually or use an automated tool, the defect rate can be calculated using this formula: Defect Rate = Total Failures / Total Test Executions Total Failures are the total number of test case failures, including failures of re-executed test cases. Total Test Executions are the total number of test case executions, including re-executions. Knowing the defect rate won’t reveal how much can be saved from finding and correcting those errors during the test phase as opposed to correcting them after deployment. The best way to determine those costs is to examine historical data related to previous releases. When deriving the costs, it is important to know who is involved in analyzing the preproduction defects—
30
• Software Test & Performance
testers, developers, business analysts and DBAs—and how much of each group’s time it takes to repair and retest them. For postdeployment, it might be necessary to add users and help desk staff to the list of those analyzing and testing defects. If you want to use a shortcut approach rather than deriving your own costs, beware! Relative costs increase as
defects are found later in the SDLC (see Figure 1). Keeping the numbers simple, let’s assume an average loaded cost of US$100 for everyone involved in correcting the defect. Using the information in Figure 1, the cost to repair a single defect is: • $500 during test • $1,500 during production Let’s go back to our example so that we can apply some of this information to determine the value of test automation relative to the increased number of defects found. Table 4 shows the number of defects that we expect to be in the application under test, based on a defect rate of 30 percent for new functionality and a defect rate of 5 percent for regression testing. Table 5 then applies the costs associated with finding the defects in test versus production to the total number of defects found in each phase. The results reveal the difference (assuming 95 percent test coverage of new functionality and 55 percent test coverage of regression functionality for manual testing, and 100 percent test coverage for both new and regression functionality using automation testing). Based on this information, automated testing offers the following benefits over the course of the four releases in our example: • 25 percent increase in test coverage • $2,304 saved in test execution costs • $19,000 saved in defect repair costs
TABLE 3: DOLLARS AND SENSE Manual Test Execution Costs
Automated and Manual Test Execution Costs
Difference
Release v.1.0
$2,300
$2,300
$0
Release v.1.0.1
$1,250
$625
$625
Release v.1.4
$2,100
$1,413
$687
Release v.2.0
$3,550
$2,588
$992
Project ID
TOTAL
$2,304
TABLE 4: EXPECT DEFECTS Project ID
Number of New Tests
Number of Defects Expected
Number of Regression Tests
(30% Defect Rate)
Number of Defects Expected
Total Defects Expected
(5% Defect Rate)
Release v.1.0
200
60
0
0
60
Release v.1.0.1
20
6
150
8
14
Release v.1.4
80
24
165
8
32
Release v.2.0
160
48
235
12
60
NOVEMBER 2006
REGRESSION TEST AUTOMATION
FIG. 1: RELATIVE COST VS. PROJECT PHASE
15
16 14 12 10 8 5
6 4 0 .2
0 .5
AAnalysis na lys i s
DDesign e s ig n
2
1 .2
0 CCode ode
TTest est
Installation Ins ta lla ti o n
DLC P hase S SLDC Phase
Benefit 4: Better Repeatability, Reliability The ability to easily and reliably repeat an automated test suite varies with the type of automation approach implemented. The five approaches are Record and Playback, Data-driven, Modular, Keyword and Database-driven. Each has its pros and cons. Depending on the size and scope of the test automation effort, an inverse relationship often exists between the ease of automated test case creation, implementation and script customization versus test case maintenance. For each of the five approaches, the implementation costs are based on three factors: the time spans allotted for test case creation, customization and maintenance. By applying a “rule-of-thumb” estimation model for each of the automation approaches, we can see in Figure 2 that over the course of the four releases, the more advanced approaches (modular, keyword and database-driven) take substantially less time to maintain and to execute than the two simpler approaches (record
and playback and data-driven). From this example, as the number of test cases to be maintained increases and a more frequent release schedule is applied, advanced approaches (which generally have higher up-front costs) more than pay for themselves over the long run. Because the more advanced approaches tend to be easier and cheaper to maintain, they actually increase the likelihood that the test group will be able to maintain the test suite and use it in subsequent releases. Those approaches also increase the repeatability and reliability of automated tests.
Metrics First! Tracking metrics for an automated test effort isn’t much different from tracking them for a manual test effort. In most cases, the same metrics are used. The key to calculating the financial success of an automated test effort lies in tracking the metrics prior to implementing the automation tool. This will establish a substantial comparison basis to measure against. Key metrics to track include:
TABLE 5: PROJECTED DEFECTS Project ID
Projected Number of Defects Found
Cost Differential
Manual Test
Automated Test
Release v.1.0
57
60
$3,000
Release v.1.0.1
10
14
$4,000
Release v.1.4
27
32
$5,000
Release v.2.0
53
60
$7,000 TOTAL
NOVEMBER 2006
$19,000
www.stpmag.com •
33
REGRESSION TEST AUTOMATION
FIG. 2: AUTOMATED TESTING TAKES LESS TIME OVERALL
4,500 4,000 3,500 Hours
3,000 u rs
2,500 2,000 1,500 1,000 500 0 R e&c Playback o rd Record a n d Pla y b a c k
D Data-Driven a ta - D r iv e n
MModular o d u la r
KKeyword ey w o rd
Data ba s eDatabase-Driven D r iv e n
Maintenance Time Customization Time Creation Time Creation T ime Customization T ime Maintenance T ime
Our “rule-of-thumb” estimation model allowed 4 to 6 hours per test case for (1) record/playback or (2) data-driven automation approaches, 2 to 3 hours for (3) modular and 1 to 2 hours for (4) keyword or (5) database-driven approaches. Allotted customization time per test case was 10 to 15 minutes for (1), 20 to 30 minutes for (2), 4 to 6 hours per module for (3), 3 to 4 weeks for the engine for (4) and 4 to 5 weeks for the engine for (5). Estimated maintenance time was 1 to 2 hours for (1), 30 minutes to 1 hour for (2) and (3), and 3 to 4 hours for engines of (4) and (5) plus 5 to 10 minutes per test case/release for (4) and 1 to 2 hours for the suite of test cases/release for (5).
34
• Software Test & Performance
• Test coverage percentage • Test execution costs • Defect repair costs (in each phase of the SDLC) • Implementation and maintenance cost for the automated solution With these metrics, the potential savings offered by automating a test effort can be established. Various test automation frameworks also can be evaluated to determine which provides the least expensive overall solution based on the number of test cases to be automated and the frequency with which the application is changed. Using the metrics in this article and your organization’s actual resource costs, you can identify the point of convergence where automation setup and maintenance costs are offset by the savings derived from improved test coverage, reduced test execution time, decreased production errors, and the increased repeatability and reliability afforded by test automation. ý AUTHOR’S NOTE While cost savings account for the primary value derived from test automation, increased revenue— through shorter development cycle times and faster time-to-market—can also be achieved. However, in many instances, increased revenue is much more difficult to calculate and therefore isn’t discussed in this article.
NOVEMBER 2006
Donâ&#x20AC;&#x2122;t Miss Out
Test & QA Report e-newsletter! On Another Issue of The
Each FREE weekly issue includes original articles that interview top thought leaders in software testing and quality trends, best practices and Test/QA methodologies. Get must-read articles that appear only in this e-newsletter!
Sign up at: www.stpmag.com/tqa
34 weeks from NOW: Hackistan04 exploits a security flaw, steals customer records and brings your system to its knees. Party’s over.
21 weeks from NOW:
MGC $168.23
BusinessMag praises the system. Wall St. goes nuts. It’s party time. MGC $89.23
12 weeks from NOW: Development ignores your concerns. The system goes live as scheduled. Customers love the new features. MGC $52.24
NOW: MGC $44.09
1 week from NOW: You raise concerns about the lack of application security during Megacorp’s major IT systems overhaul. MGC $39.51
35 weeks from NOW: Customers, shareholders and employees call for your head. But you’ve got a plan to make them all go away. MGC $8.17
Wouldn’t you rather know NOW? 75% of hacks are made possible by software vulnerabilities. That’s why Fortify strengthens applications at the source: the code itself. What’s that mean for you? A little more sleep at night. While the hackers take the plunge. Visit fortifysoftware.com/10Q for 10 questions you’d better ask to be sure your company’s assets are secure.
>> fortifysoftware.com
The Securit y Zone
Application Security & Vulnerability Testing
Photo Illustration by The Design Diva, NY
Scan Your Source Code To Locate Weak Spots Early By Joe Basirico hite-box testing is notoriously difficult to do. Without automatic code-scanning tools, it takes a keen eye, concentration and an enormous amount of time to scan each line of source code for security vulnerabilities. And as hackers become more sophisticated at finding these chinks in your armor and writing exploitative code, it becomes more necessary to take every precaution before shipping software. The necessary protection can range from security training throughout the security development life cycle (SDLC) to using tools such as source code scanners and vulnerability scanners. In 2005, according to recent CERT/CC statistics, more vulnerabilities (5,990 in total) were reported in shipping software than in any previous year—topping the previous year’s total by more than 2,000 (see www.cert.org/stats)—and this year is heading in the same direction. Despite attempts by the larger and more proactive players in the industry—Cisco, HP, IBM and Microsoft—to incorporate security in all phases of the SDLC, the number of reported vulnerabilities continues to increase. The increase in security vulnerabilities leads to several questions: Where are the greatest weaknesses in software security? Were over two thousand more vulnerabilities discovered because security trainers didn’t transfer their knowledge properly? Or are hackers getting better at discovering these vulnerabilities? Do software corporations rely too much on perimeter defenses, such as firewalls, intrusion detection systems, deep packet inspectors and antivirus solutions? Is it just sheer complacency? Or is it really a combination of all of these things and more? The final and most important question is one we can all consider and help to answer: What can we do next to ensure that we ship fewer and fewer security vulnerabilities in the subsequent versions of our software?
W
NOVEMBER 2006
All Security, All the Time The answer? We should train every member of the team to think about security in all phases of the SDLC. From the moment that project managers start talking to customers and gathering requirements to the final shipping build and deployment, security must remain in focus. This is a difficult task—most software teams are already stretched thin, with more demands on their time than hours in the day. And tightened budgets for development and testing teams make purchasing new tools, even when legitimate, a greater challenge. One tool that is invaluable in the development phase of the SDLC is the static analyzer. This tool can discover elusive programming errors before runtime, when they become much more difficult to find, as well as many logical and security errors in an application before compilation. When choosing static analysis tools, consider their individual benefits and limitations, as well as the specific types of bugs they’re good at finding.
Status Analysis Tool Success Microsoft has learned how to make static analysis tools work to its advantage, putting every major project through rigorous testing with static analysis tools at both check-in time and at the Joe Basirico is a security analyst with Security Innovation. With a focus on incorporating security into the SDLC, he also serves as a conference speaker on software development best practices. Contact him at jbasirico@sisecure.com. www.stpmag.com •
37
The Sec urit y Zone final build. Each code modification is checked on the developer’s computer and in the context of the complete solution before the nightly build. Microsoft has developed its own internal static analysis tools called PREfix and PREfast. These tools have evolved over the years and are adept at identifying both security flaws and functional problems before build time. The ability to run static analysis on all of the Windows source code is a testament to the maturity of these tools. NASA also requires that every code change to mission-critical applications go through static analysis. Every warning and suggestion the analyzer finds must be either fixed, or a comment must be added to the source code explaining why the warning or suggestion required no change to the code. Most major software companies employ some form of static analysis tool; in fact, it’s clear that the exercise of these tools helps you to deliver a more robust product at ship time.
•
be used, it can easily warn the developer of cases when uninitialized variables could cause problems. These problems often manifest themselves as “divide by zero” errors or other improper function calls. Many modern static analysis tools attempt to match memory allocations with deallocations. This helps uncover many memory leak problems where a developer forgets to deallocate small bits of memory that, over time, may build up and cause the system to become unstable or develop a security flaw. Memory leaks are notoriously difficult to find at runtime and can require many hours of repetitive, pedantic testing. Logical statement checking can help the developer or white-box tester to discover cases in which an if or while statement may always evaluate to the same result. Often, logical statements can be built up, version after version, until simply getting one correct output is enough. Developers sometimes get it working; then step slowly away, trying not to touch their if statements with a slew of ands and ors. These undertested if statements may have fallen into the case when no matter what inputs are given, the same outcome occurs. Static analysis tools can quickly evaluate these statements by simulating all possible values of each variable and discovering which logical statements should be redesigned. Security exposure may be reduced by removing unnecessary libraries from the application’s include sttements. Static analysis tools can quickly check which libraries or APIs are required by the application and alert the developer to remove them. Since we inherit security vulnerabilities from each of the libraries we call into, removing unnecessary libraries is a security best practice. Many developers and testers fail to test for buffer overflows in their application because they think they’re safe due to the use of a managed application written in .NET or Java. Because many of these applications call into legacy code that was written long ago and contains other security vulnerabilities, removing all unnecessary libraries can shrink the attack surface. Many other security checks can be found at compile time. Some system API functions have been deprecated and should be replaced with their safer cousins. A static analysis tool can quickly discover the use of any of these functions and notify the developer of a more secure replacement. Possible buffer overflow conditions can be predicted so the developer can fix them before they’re discovered and exploited through the application. Discovering buffer overflows at this stage also shows the developer the exact line and conditions when a buffer overflow may occur, which can be very helpful in finding and fixing the vulnerability. Time-of-check versus time-of-use problems can be discovered by showing places where the developer has checked the
No single tool or practice can possibly discover or fix all security vulnerabilities.
State of the Field The field of static analysis tools is becoming heavily populated and very competitive. Companies like Klocwork, Fortify and Compuware have created some interesting and diverse offerings for static analysis. The techniques used by static analysis tools are often guarded as a company’s key intellectual property; simple pattern matching is not enough. Some of the techniques employed by static analysis tools include: • Semantic analysis • Strong type checking • Memory allocation checking • Logical statement checking • Interface and include problem checking • Security checking • Metrics • Simulation Semantic analysis allows the analyzer to discover the basic structure and relation of each method within the application. The static analysis engine can then build a syntax tree that the simulator will use later to calculate how the application will execute at runtime. This allows the static analyzer to find bugs deeper in the application by traversing the syntax tree for method context that approximates what would occur at runtime. Strong type checking helps prevent dangerous type-casting assumptions such as trying to cast a decimal to an integer and losing precision at runtime. This can cause many security vulnerabilities in memory allocation and rounding. Remember the adage “1 plus 1 may equal 3 for very large values of 1.” Within the strong type checks, the static analysis engine also ensures that each variable is properly initialized before use. Since it can quickly discover how and when the variable is to
38
• Software Test & Performance
•
NOVEMBER 2006
The Sec urit y Zone availability of a resource and allowed a significant amount of time to pass before using that resource. Time-of-check versus time-of-use errors can allow escalation of privilege attacks, which may allow tampering with a resource to which the user shouldn’t have access or information disclosure. Metrics can help a developer locate unnecessary complexity in a piece of code. This can significantly increase software performance, which reduces the possibility of a denial of service attack. Static analysis tools can predict which functions are very complex using cyclomatic complexity, functional file coupling and functional file cohesion. Other metrics, such as the ratio of commented lines to source lines, can point to places in the source where the code may need to be commented more effectively. The simulation engine is one of the most powerful and most guarded pieces of the static analysis tool, allowing the tool to predict how the application will behave after it’s been compiled and run on the system. In a simulation engine, the simulator selects a function, generates data for each of the variables, and runs the function through every possible code path. If information about proper data ranges for those variables exists within source code, it will be used. If no information exists, random values selected between the max and min for that data type are used. Testing with a static analysis simulation engine is a good way to get around the inherent shortcomings of traditional static analysis tools. In its final check, a static analysis tool can crawl the source code to map out every possible code path and discover unused or unreachable code. These dead code pockets are orphaned and unreachable, so they remain untested throughout the product cycle. In a future release, bug fix or code modification, the dead code may become active, thereby exposing untested security vulnerabilities or other functional flaws. Under certain circumstances, a skilled hacker may even be able to circumvent the built-in constructs within an application and execute the untested code, which may then contain exploitable security flaws.
consuming to find by other means, including: • Buffer overruns – These vulnerabilities are eye candy to a hacker. They are often exploitable and may allow the hacker to take complete control over the system. Buffer overflow conditions are difficult to spot in manual code inspection because multiple conditions may be required to trigger the error. • Least privileges – A process should always run with least privileges, ensuring that if the code is exploited, the undesirable code is given limited power. • Dangerous functions – Some system functions may open your application to possible security flaws; these should be examined and replaced with more secure functions (for example, replace strcpy with strncpy). • DACL problems – A null DACL gives no protection and is a warning sign that an object being used by the application is not as secure as it should be. • Canonicalization problems – There are many different ways to represent a file, URL or device. A hacker may be able to gain access to a protected file by using alternate representations of the filename. • Exception handling – If an exception handler isn’t present, the application may terminate or be left in an unpredictable state when problems occur. • Format string problems – Functions such as printf, scanf, sprintf and others may open an application to problems
Kill Bugs, Save Money Employing static analysis tools during testing can drastically reduce the number of bugs that make their way to the blackbox testing phase. These tools can be executed quickly, and the scanning is easy to do before source code check-in. While not often considered, one of the first things a hacker attempts when compromising a system is to steal source code. Source code allows the hacker to quickly discover vulnerabilities in the application at the source level by using security scanning tools. This low-hanging vulnerability should be eliminated before shipping, or it may open a large attack surface if the source is compromised. With the gaining prevalence of Java and .NET applications, decompilers also give the attacker a significant edge to discover vulnerabilities with source code scanners. Scanning tools can help reduce development and testing costs because they catch problems early in the SDLC. Bugs are expensive to fix later on, and security vulnerabilities are infamous for being elusive. Source code scanners are useful assets because they discover vulnerabilities that are difficult or timeNOVEMBER 2006
www.stpmag.com •
39
The Sec urit y Zone when user input is interpreted as the format string. This can lead to an exploitable vulnerability that may allow an attacker to execute code remotely. • Input validation – A hacker may be able to cause complete system compromise if improper input exposes a buffer overrun or format string bug. • Ignored return vvalues – Ignoring return values can result in a variety of reliability and security bugs that can be difficult to debug and reproduce. • Package insertion – Package insertion can allow untrusted code to run in the context of a trusted Java application and may therefore spoof or otherwise attack the user. • SQL injection – SQL injection is a technique used by hackers to probe databases, bypass authorization, execute multiple SQL statements, and call built-in stored procedures. Because they are the most popular vulnerability types for hackers, each of these common vulnerabilities must be scanned for and discovered.
No Panacea
• Software Test & Performance
Throughout the Product Cycle Since static analysis tools are employed before build time, they can save time, money and retesting aggravation. The cost to find and fix a bug rises drastically throughout the product cycle: If a bug is found by the customer after the application has been released, it can cost literally millions of dollars. Since static analysis tools are run early in the product life cycle and can point out the specific line number where a programming error has been made, the bug can be found and fixed very quickly and inexpensively. Advanced static analysis tools not only show the line number of the bug, but also display the code paths that lead to its execution. Static analysis tools can and should be employed throughout the product cycle. Developers can use a lightweight version of the tool to check for simple bugs that may have been missed at development time. Build managers or lab technicians should use the tool to isolate the more sophisticated bugs at code integration time. After a build is created, you can use static analysis tools to ensure code coverage and to discover complex sections of the product that should be tested more thoroughly. Developers can run a stripped-down version of the static analyzer on their local build machine before check-in to enforce coding standards and readability, and to check for security bugs. Enforcing coding standards and readability will ensure that code is easy to update and maintain in future releases. Many security bugs such as buffer overruns, input encoding issues and use of dangerous functions can be found before the source is built into the complete shared source tree. Once the developers check their code changes into the source tree, the build manager can run the static analyzer to check for integration bugs. In this step, the build manager can enable options that may not have been available to the developer, such as simulations, function integration options and dependency diagrams. Simulations may require some
Use multiple tools, training and best practices to ensure your software is as secure as possible.
No single tool or practice can possibly discover or fix all security vulnerabilities. Instead, multiple tools, training and best practices must be employed to ensure that your software is as secure as possible. Some of the limitations of static analysis tools include false positives, vulnerabilities that show up only in the environment, and the tendency to give a false sense of security. False positives can be difficult to weed out and often require significant security experience to discern which warning should be fixed. Fixing every warning reported by a static analysis tool can cause unnecessary code churn, which may lead to the introduction of more functional and security flaws. False positives that aren’t easily interpretable can overwhelm a developer to the point that he or she becomes disenchanted with the tool. This plays into the tool’s psychological acceptability; if it’s too difficult or causes significantly more work for the developer or tester to use, it may be discarded, despite the potential of increased security in the end product. Many vulnerabilities occur only in specific environments and will be discoverable only on certain install beds. These vulnerabilities are often caused by low system resources, library and API versioning problems and insecure settings. These types of vulnerabilities are impossible to detect before runtime. Static analysis tools attempt to simulate resource contention and some versioning problems, but the vulnerability can be detected only at runtime with dynamic analysis tools such as a debugger or fault injection tool. Library and API versioning problems are difficult to detect and test for before runtime; however, with strong naming conventions, a developer can take precautions to ensure a certain library has been
40
•
installed on the system. This approach, while secure, is sometimes unused because testing on every available system is difficult and costly, and limits the number of compatible systems the application can be installed on. Static analysis tools can’t foresee possible insecure deployment settings and thereby miss this significant attack surface. Ensuring the product ships securely and is installed securely by default is challenging and must be outlined in the requirements early on. It is essential to understand: • The environment your software will be deployed in • How your users will use your software • How knowledgeable your users are • How much configuration is likely to occur after the product has been installed on the system Once these questions have been answered, secure requirements can be written to help keep your users safe in many deployment situations.
•
NOVEMBER 2006
The Sec urit y Zone external states to be set before running the newly added code, which will be available only with the complete source tree. At this point, the build manager can send bug reports back to the developers so they can fix code integration errors before they make it to the build. The two-sided analysis of developer and build manager will help remove many bugs before the source is built, significantly reducing development and bug repair costs. After the build is complete, you can use static analyzers to discover areas of higher cyclomatic complexity that will provide insight about where deeply hidden bugs may lie. Using the metrics and complexity features of a static analysis tool, you can discover code paths to execute the complex function at runtime. Once the code path has been found, you can use that information to execute the complex function from within the built application and discover complex runtime errors.
Attributes to Look For There are many excellent static analysis tools available on the market today. Many of them offer great features and promise to secure your application from unique and isolated threats. Matching the right static analysis tool to your specific development process can mean the difference between a painful disaster and a seamless integration with your current development process. The most obvious requirement for any static analysis tool is to ensure that it integrates well with your current development environment. Whether it’s a plug-in for Eclipse or an add-in for Visual Studio, make sure that it works well and doesn’t require too much hardware overhead, such as extra processor power or memory. It’s no fun to be forced to upgrade all your development machines just to get a new static analysis tool to work. Also, make sure that the static analysis tool you choose supports not only the programming languages you’re using currently, but any anticipated languages as well. Many static analysis tools are definition based, which allows you to update them as new vulnerabilities are discovered. This great feature can save you a lot of time and money down the road. Also, look for a static analysis tool that allows you to write your own definitions. This will enable even tighter coupling between the base static analysis tool and your internal best practices. Ask your current development team if they have suggestions for static analysis tools that they’ve found to be effective. Ramp-up time on these tools can be significant, so if there’s a developer who’s already experienced with a tool, it may be worthwhile to use that developer as a resource when deploying the tool. If you don’t gravitate toward a specific static analysis tool, consider the overhead time it takes to train the development team on each tool, and the support that will be provided by the static analysis tool company.
up projects specifically designed to test the tool to a full test of the whole project for a short amount of time. If you choose the latter, prepare for a lot of overhead time as the developers learn a new tool. When deploying a new tool, be sure to appoint a dedicated team member who will master the tool. This person—a well-trained and competent developer—should understand all the features of the tool and how to apply them to the current project
Beyond Tools, Security Is King For optimal effectiveness, static analysis must be coupled with security awareness training for developers so they can understand the threats and readily interpret and act on the results. Secure coding courses range from classes that assume no previous security knowledge to advanced courses that challenge developers to understand secure development practices completely. At any level, make sure the source of your training is well-qualified and knowledgeable—many self-styled “experts” are ready to pass themselves off as application development, testing and security gurus. Once you receive training and feel knowledgeable in the field of application security, remember to tap into the services of external security testers to help audit and gauge your performance. By keeping security in the front of your mind throughout the development process, you’ll ensure more secure products that will delight your customers. ý
Start Small Before diving into the use of static analysis tools, consider deploying a few on small projects to get a feel for their usability and capacity to integrate with tools already being used in your development shop. These small projects can range from small mockNOVEMBER 2006
www.stpmag.com •
41
Best Prac t ices
I’ve Seen the Future: All Services, All the Time Testers, beware: The New is at least a crude baromeYork Times is now writing ter of mainstream technolwith some regularity about ogy trends. And suffice it Web services. This slow say, the pressure to implemove into the mainstream ment Web services appears business and tech lexicon to be rising. means new challenges for Web services typically are those in testing and QA built upon service oriented roles who actually have to architectures, or SOAs—yet make this all-services, allanother in the litany of the-time future come true. hype-laden and obscuring Geoff Koch Consider a summer spate acronyms that litter the of services-related stories in the Gray Lady. technology landscape. For purposes of In a May 8 article about IBM’s new this column, let’s just agree that SOAs software tools to make its mainframe are built from loosely coupled, reusable business more Internet-friendly, Times and network-based services. In short, reporter Steve Lohr wrote, “Web servSOAs should make it easier than ever to ices technology makes programs more patch together different programs. flexible, as individual building blocks of But, for at least two reasons, maybe code will be able to communicate with they don’t. others automatically.” Simple,Yet Complex Nearly a month later came news of First, despite the endless paeans to wellthe launch of Google Spreadsheets. defined and platform-independent “Google is clearly hoping that the servinterfaces, SOA-based applications are ice… will give it a head start on Microsoft flat-out more complex than those that in the area of so-called Web services,” interact with the wider world by way of wrote John Markoff on June 6. graphical user interfaces, or GUIs. A few weeks after that, it was Damon Messages passed from service to service Darlin, a recent émigré from Business are designed to conform to mysterious 2.0 magazine to the Times, writing about enterprise architecture and often look the migration of many common software like gobbledygook to even the wellapplications from the PC to the Web. trained eye. New software tools “enable a host of so“If you look at today’s systems, I think called Web 2.0, or Web services, appliit’s much more the norm to have a set cations like Google maps posted in Web of headless processes,” says Bob sites or photos displayed on Flickr.com,” Carmichael, CTO of Solstice Software, wrote Darlin on June 17. based in Claymont, Del. “With asynMentions of Web services now crop chronous communication, that’s just up several times per month in various sort of the de facto architecture.” Times articles. In contrast, last year the In the good old GUI days, testers phrase appeared 22 times. In 2004, just would input various data in custom 11 times. screens and then sit back and watch for At this point, any tester worth his or the application to throw an exception. her salt should be screaming, “Who Beyond text editors and a rudimentary cares?!” Though arguably the world’s knowledge of HTTP to send and receive most powerful newspaper, the Times is messages, the most important tool of hardly at the leading edge of software the trade was a sharp eye—something news. But as a database, nytimes.com NOVEMBER 2006
that’s of little use in a world where XML responses can look startlingly varied and still be correct.
SOA Sprawl The ability to anticipate and monitor usage is another SOA challenge that weighs heavily on testing organizations. Parasoft’s Wayne Ariola recalls a recent meeting with a heavy equipment manufacturer experimenting with SOA and Web services. The manufacturer estimated it had more than 80 internal services marked mostly by their lack of centralized control and limited visibility. “One of the primary benefits of SOA and Web services is reuse, so quite honestly, your greatest risk can be too much success,” says Ariola, vice president of corporate development product management for the Monrovia, Calif.–based software automation tool vendor. “We call this SOA sprawl.”
Requirements: Revise and Refine Even before SOA and Web services, the most wallet-draining aspect of developing software was achieving a true understanding of an application’s requirements. Business users don’t always have a good grasp on the technology or its possible benefits, and sometimes struggle to say just what they want the software to do. “Now, with SOA, the challenge is even greater, and the requirements are even vaguer,” says Chris Houck, a vice president of product marketing for OpenConnect Systems in Dallas, Texas. Houck estimates that 70 percent of Best Practices columnist Geoff Koch writes about science and software from Lansing, Mich. Though he knows more about journalism than AJAX, he’s interested in Web 2.0 technologies. Write to him at koch.geoff @gmail.com with feedback on his column, and he will respond. Now how’s that for service? www.stpmag.com •
43
Best Practices project budgets is spent revising and refining requirements. Half of this money is spent up-front in analysis and design, while the rest goes toward testing, where Houck says the “real requirements become painfully obvious.” Of course, vendors galore are happy to offer their wares to address these and other SOA-related testing travails. Indeed, many of today’s tools help testers in either instrumentation or emulation tasks—important capabilities in an era of sometimes mindnumbingly complex combinations of hardware and software.
XML, XPath/XQuery and BPEL. You may also want to sidle up to those business types who understand the wider context of what you’re working on. Even though the MBA crowd still might be
•
and shareable business services. “Testing Web services/SOA applications involves validating messages that are flowing across business services,” says Fred Holohan of Active Endpoints, a Shelton, Conn.–based provider of SOA technology. “Understanding the correct content and context of those messages is critical to validating proper execution and exposing subtle problems.” Just when will the servicescentric future arrive? Soon, though probably not tomorrow—at least based on the venerable Times. As of the eve of this column’s September 15 deadline, a search of nytimes .com for “Service Oriented Architecture” returns 0 results. So at least for now, maybe you can hold off getting smart on SOAP and tagging along with the suits on their weekly sushi outings. But the Times is covering AJAX, so you have something to work on in the meantime. ý
The future is in grokking SOAP, WSDL, XML, XPath/XQuery and BPEL.
Back to Basics— and Business But before spending money, you first may want to spend time learning the basics of Web services—and SOA-related technologies. Sorry to those who still practice GUI-centric testing, but the future is in grokking SOAP, WSDL,
• fuzzy on just what services technology is capable of, they likely have a clearer idea than you about the looming market opportunities or competitive threats. Forget coding and testing in isolation. The goal today is to encode nuanced
Index to Advertisers Advertiser Agitar Axosoft Critical Logic Enerjy Entrek Fortify Software IBM Instantiations ITKO Mindreef Pragmatic Software Seapine Software Softlogica SpiDynamics SQS Software Security Summit
URL www.agitar.com/learnmore www.axosoft.com www.critical-logic.com/products/TMXwrx www.enerjy.com/visible www.entrek.com www.fortifysoftware.com www.ibm.com/takebackcontrol/flexible
Software Test & Performance Test & QA Report
www.stpmag.com www.stpmag.com/tqa
Worksoft
www.worksoft.com/stp1106
44
• Software Test & Performance
Page Number 31 2 47 3 34 36 24-25
www.instantiations.com www.itko.com/lisa www.mindreef.com/quality
8 33 42
www.SoftwarePlanner.com/STPA www.seapine.com/stptcmr
34 4
www.LoadTestingTool.com www.spidynamics.com/QA1 www.sqs.com www.S-3con.com
39, 41 48 32 11 45 35 6 NOVEMBER 2006
Future Future Test
Test
Successful SOA Surprising as it sounds, cessful and reap the necesmost service-oriented archisary ROI, development tecture (SOA) projects are teams need to take an agile untestable. In nearly every approach to testing, using case I’ve examined, the a collaborative testing tool business and project team for each and every step of can’t see what’s truly going the software development on under the covers of life cycle (SDLC). their SOA. In short, successful SOA Worse yet, large organideployment requires autozations with complex intemated end-to-end testing. Chris Benedetto gration projects typically From Infancy to Delivery invest four times the necessary labor, time To prevent getting left in software innoand capital to validate their SOA. And even vation dust, testers, developers and QA then, it can fail in production. Why? professionals need to tap into complete The Right Tool for the Job SOA testing—starting today. They can As the number and breadth of SOA accomplish this by: • Using automated testing tools. The implementations increase, quality is more integrated the system, the emerging as a significant challenge. more complicated functional testThat’s because the service-oriented ing becomes. Software development architecture is a new and different enviteams need to use automated, endronment with a dynamic, real-time, to-end testing to look deep inside always-on infrastructure—vastly differthe complex infrastructure of an ent from what companies are used to SOA, and rapidly and effectively with traditional GUI architectures. find and correct hidden defects As such, manual and traditional testbefore they go into production. ing tools just don’t cut it when it comes • Bridging the technical divide. to testing SOA—it’s like trying to fit a Software development teams need square peg in a circular hole. They just integration tools that bridge the aren’t the right tools for the job. technical divide between the techIndeed, developers’ frustration with savvy developers and testers, and SOA usually has less to do with the techthe not-so-tech-savvy QA teams and nology itself, and more to do with what business analysts. When everyone SOA teams nearly always lack: the can view and understand the variability to automatically test, debug and ous phases of the SDLC, they can validate. intelligently compare their assessSOA can be an efficient and valuable ments, understand exactly what business asset—and more and more went wrong or right, and know how businesses are recognizing that every to move forward with production. day. The ease and low cost of reusing • Starting integration and SOA testing application components to create new earlier in the SDLC. SOA needs to systems, coupled with the ability to use be tested from infancy to delivery. open standards such as XML, SOAP and This way, defects can be detected so on make SOA easy and cost-effective as soon as they rear their ugly to implement. heads, and corrected before they But SOA demands a highly strucwreak havoc on other components, tured quality testing approach that production schedules and operatincludes continuous testing, validation ing budgets. and assurance of all processes across all • Creating simulated test cases. To protocols, messages, languages and servbegin SOA testing early, testers ices. For SOA deployment to be suc-
46
• Software Test & Performance
need to simulate unavailable systems, instead of waiting for all the components to be developed. As systems come online, testers can simply turn off the simulators. With simulation, there’s no excuse for not testing early.
As SOA Grows… Looking forward, I predict that SOA implementations will hit peak maturity in 2008. That’s when I see the practical tools and standards to build SOAs fully developed and available, and the supply and demand for SOA equalizing. Clear governance standards need to evolve and mature to make SOA testing quicker and easier to implement. Standards keep all vendors and end users operating on the same wavelength, so unified testing and quality assurance can support an organized and efficient SOA. Today, organizations like OASIS, Open SOA, SOA Link, and WS-I are forming to discuss models and specifications. To date, however, no firm standards or governance models have been widely accepted across the testing industry. Without consensus on SOA testing and quality standards, SOA adoption will stall, vendors will take wildly different approaches and end users will ultimately suffer. We need governance models that specify normative definitions of services and establish how methods and protocols will address services. Likewise, as services are hosted in registries, we need automatic ways to initiate testing of these services and underlying components. As we shift a 25-year computing and programming model from a monolithic application style to a completely distributed, heterogeneous, federated mode based on standards-based connectivity, we need to know that the changes we implement are working as we expect, and with the quality we require. Today, end-to-end testing can facilitate successful SOA deployment, and tomorrow, end-to-end testing will be even more useful and easier as consensus around standards evolves. ý Chris Benedetto is a vice president at Solstice Software. Write to him at cbenedetto @solsticesoftware.com. NOVEMBER 2006
THE SMALLEST
SECURITY DEFECT COULD BE THE START OF A
BIG PROBLEM Find Security Defects with QAInspect
QAInspect is a unique product that gives QA professionals the ability to incorporate automated Web application and Web services security testing into your test management process quickly and easily without impacting schedules or requiring security expertise. • Extends existing functional tests with comprehensive security testing • Includes testing and reports for more than 20 major laws, regulations and best practices • Reports and prioritizes security defects alongside functional defects • Seamlessly integrates with IBM Rational Software Development Platform (ClearQuest and Functional Tester) • Seamlessly integrated into the following Mercury products: • Quality Center • TestDirector® • Business Process Testing • QuickTest Pro • Winrunner TM
TM
TM
TM
FREE Trial Offer: Download a FREE trial and learn how QAInspect simplifies security for QA Professionals at www.spidynamics.com/QA1. Toll-Free: 1.866.774.2700 QAInspect is a trademark of SPI Dynamics.
TM
Start Secure. Stay Secure.