autodesk insiders the comparisons to the saved images will fail. This test will now run automatically, and save us from having to regularly manually check the tested functionality, giving us more time to test the new stuff. Image Capture tool? I don’t remember seeing that in Revit. Do all regression tests use it? Image Capture is one of the many tools that we’ve created for our testing, diagnostics and repair. These tools exist in the Revit code, but can only be accessed via a special in-house debug mode. Some tests use Image Capture comparisons, but others measure performance (with timing benchmarks), compare memory usage, compare Ribbon/Application Frame states, evaluate various export formats, etc. Also, we often manually add lines to a journal (using standard VBN scripting language) which will make it do things that would be very difficult, and time consuming, to do by hand in a Revit session. For example, we can draw a single wall in Revit and move it a few feet. In the journal, we can find those two steps and add an If-Then-Else loop that will make it replay it to create 1,000 walls… or 100,000 walls! Isn’t bug-filing your main responsibility? A main goal of QA is to help Autodesk deliver high quality software that has the fewest bugs possible—and finding the bugs is probably the most important aspect of our job. But we ultimately want to get the bugs fixed (hopefully before we ship the software to our unsuspecting users), so filing them skillfully, and becoming bug advocates, is also imperative. In terms of actually operating the software as users, our Revit developers have a wide range of Revit knowledge. Some developers are proficient users, others are expert only in the one or two feature areas on which they’ve worked, and the rest are on a spectrum in between beginner and expert. What this means for QA and bug filing is that clearly communicating the problem is a key aspect of good bug filing. We want the developer to quickly understand what the problem is and exactly how to reproduce it. The developer’s time is precious and journal playback with their debugging tools can be quite slow. So, the more efficient our journals and descriptions are, the better it is for our developers… and also for our Autodesk customers. A bug report (SPR, in our Revit acronym speak) has several important parts: SPR Summary - This is a frequently searched field in the Revit database (which we call the SRD). Before we file any bug, we check here to see if it has been reported before. The Summary must include any important key words, and typically has an indication of the severity—especially if it involves a crash or data-loss of some kind. SPR Description -We typically begin by noting the Revit flavor and build number in which we found the bug. We 40
www.autodeskcatalog.com/AECEdge
then provide clearly numbered steps to reproduce the bug, as though the reader was a novice Revit user. If the bug involves a particular file or journal enclosed with the SPR, we’re specific and explicit about what to do with which files—as more files are likely to be added during the life of the bug. At the end of the steps, we note the observed bad behavior, along with what we expected Revit to do. Other - The SRD includes fields for lots of other searchable data, including Function, Project, Severity, Priority, Customer, Target Version, Assigned To, Reported By, Dev Status, Ship Status, Comments, and lots more. Sounds very involved, how do you find all the bugs in the first place? Lots of ways, and this is where it gets especially creative and interesting. We have to find as many of the most important and riskiest bugs as we can, as fast as we can. We have lots of powerful methods for doing this, including: Exploratory Testing -This encompasses much of what we do, and it overlaps with most other testing techniques. We learn about Revit, its usage, its risks, and the ways in which it has failed in previous testing. Exploration involves continuous learning, experimentation, backtracking, and taking a new path, for example. It’s definitely more free-form than executing tests from a script, but it often catches critical bugs. Feature (Functional) Testing - QA’s will typically be on the feature team from the early stages of design. When the developers have a prototype or other test worthy piece of the functionality, we hit it hard. Using our test plan as a guide, we test as much of the feature as we can, often creating new regression tests along the way. System Testing - In addition to responsibility for new features, most Revit QA members have specific existing features which need to be tested with respect to the new tools. This provides some important overlap with feature testing—originating from the existing tools side. Boundary Testing - This method involves pushing inputs to the limits with their largest and smallest possible values— and then going even a bit larger and smaller. Does this crash Revit? Sometimes! Ham Sandwich Testing - We occasionally do some random crazy stuff, such as trying out scenarios with various key combinations. This seemingly haphazard mashing of keys has been compared to hitting the keyboard with a ham sandwich. After ruining several keyboards we now only use metaphorical ham sandwiches! Alpha and Beta Testing - Many of the Revit customers reading this interview have probably participated in an Alpha or Beta. When most of the major bugs have been found and fixed, we then give the not-quite-ready-for-release software to several hundred Revit users to pound on for a few months. QA manages the feedback in the Alpha and Beta forums, and our resourceful customers do find some amazing bugs. Bug Bashes and Wacky Days - These are quick guerilla sessions, often lasting one-day or less, where everyone in QA launches a fast and furious attack on a single new functionalspring_2010