Trusting Instincts Is Not a Sin— Experimental Design Can Help
By Yogananda Jeppu
hildren are born true scientists. They spontaneously experiment and experience and experience again. They select,
C
combine and test, seeking to find order in their experiences: “Which is the mostest? Which is the leastest?” They smell, taste, bite and touch-test for hardness, softness, springiness, roughness, smoothness, coldness, warmness: they heft, shake, punch, squeeze, push, crush, rub and try to pull things apart. –R. Buckminster Fuller Through his experiments in the natural world, Italian physicist Galileo Galilei came to realize that nature didn’t operate as defined in Scripture: “Thou hast fixed the earth immovable and firm.” He raised this problem report and was pulled up by the Management for his views. Under pressure, he had to bow down—and sadly, was not around to say “I told you so” when the system failed. How many times as a tester have you faced this problem? You’re sure, with a strong “gut feeling,” that there is a problem. But you don’t have conYogananda Jeppu is a scientist at IFCS Aeronautical Development Agency in Bangalore, India.
28
• Software Test & Performance
crete proof due to lack of time and inadequate testing—and you didn’t have the heart to say “I told you so” after the catastrophe. Scientists have unravelled the mysteries of nature through experimentation. They may have had an original theory but would still have to prove it correct through experiments. In science, “Theory guides. Experiment decides,” the saying goes. Like nature, software today is a chaotic, mysterious object. One doesn’t know what the Creator has put into it. It’s for the Tester to dig deep into the mysteries to find out what’s hidden in its myriad loops. Experimenting with software is surely the way forward for the test team, but the lack of time makes this experimentation a costly endeavor. Numerous techniques exist to help tame the chaos, including the orthogonal array, a systematic approach that incorporates statistics à la R.A. Fisher and his Design of Experiments (DOE)(see “DOE and Taguchi”). I’ve tried orthogonal arrays and found that while they do reduce the test sizes, they’re not the end of the affair. Experimental design also can be used to further optimize the test cases to increase coverage.
Taguchi Test Flight For a sample problem, let’s test a safety-critical component of flight control software. The Taguchi approach (see “DOE and Taguchi”) can be used for other less-critical software as well, since the underlying concept of software testing is universal. So, let’s get off the ground. We’re testing the computation of the scheduled gain of an embedded controller. In fighter aircraft, the feedback controller is designed by tuning the feedback gains so that the aircraft performs as required. This has to be done at various operating altitudes and speeds to ensure uniform performance over the complete operating range. The control designers ensure this by scheduling the gains using altitude and speed as the input parameter. Altitude and speed are provided by a complex system known as the Airdata system. The gains, around 30 or more, are provided as lookup tables that are a function of speed and altitude. Intermediate values are computed using an interpolation algorithm. The tester has to ensure that approximately 5,000-odd gain values have been encoded correctly, and that the interpolation works satisJUNE 2007