Testing quality

In the past when software was small, only few hundred lines of code then testing was relatively easy. As a software developer, what I did was to make sure that the algorithms were right and analyzed the program construction to make sure that it compiled correctly. If I had errors, I fixed them then re-compiled so testing was not an issue. However, as software size getting larger, I began to see that I had several defects after released to the customer (One was overflow stack) so I started to be more careful about testing. I began to write test cases to make sure that the program ran correctly and meet the requirements with input data and output data (Black-box testing). Eventually, many programs that I wrote have to be integrated with other programs so as a team we divided our works into separate functions and features based on structure analysis method but we were still working independently, only discussed with each other when needed. Each person did its own tests and we were confident that everything should work the way we think but in the end it did not worked well. There were so many defects in our final product so my managers decided to form a testing group consisted of many newly hired programmers who were given the job of checking softwares to make sure they worked before they were released. As software developers, we only have to focus mostly on design and code. We tested our own code and not worried much about the rest of the final software. So our team consisted of two groups, developer group and testing group and this concept had been adopted in many software companies and it still is the way many companies operate today.

Now looking back for so many years my biggest surprise is how many people still believe that you can “test quality” into a software product. Today most software are very large and complex, many of my projects have about 5 to 20 million lines of code and it is impossible to test everything so instead of focus on testing, we focus on a disciplined approach to “build in quality” as we work. This is also a major difference between software engineering and computer science training. With software engineering, you must focusing on the process that creates the product and “build quality into the process” so the final product will have quality. Of course, nobody is perfect so you must rely on others to support you and identify your mistakes so you can fix it. Therefore, software reviews, code inspections, pair-programming, phase reflection, and lesson learned are applied to ensure that we will have quality product.

When students come to my class at CMU, many have an attitude of “Why cares about defect when we can fix it anyway” or “Code first, fix defects later” so I tell them a scenario: “Assume that you are flying in an airplane and you hear the pilot said that he has software defect in the control system and must “reboot” the system. How long will it take to reboot the system? Only ten or twenty minutes and that mean the airplane will have no power during that time and of course will crash” and I ask: “Do you really care about software defect or not?” As the class laughing aloud, I add: “I am sure it depends on your point of view, you probably care if you are on the plane and do not care if you are on the ground. So the real issue is whether defect matter in a theoretical sense but whether it matter to you. Let me give you another example, you are the owner of a software company and you have hundred software developers working for you. If customer finds software defects in your product then you have to fix it. The cost of discovery, recovery, reporting, repairing, redistribution, and reinstallation costs for every defect would average about $ 4000 each and there are hundreds or thousands defects for every product that you sell. Do you really care? How much do you sell the software for and how much you have to spend on fixing defects? Regardless of their causes, defect costs are very expensive, if you had to pay this cost, you would really care about defects. My question is, as the owner of a software company, do you still hiring people that do not care about defects?

According to the research of Professor Watt Humphrey at Carnegie Mellon, experienced software developers inject one defect in about every 10 lines of code. While these numbers vary widely from one software developer to others, and they include all the defects, even those found in reviewing or by the compiler, there are still lots of defects. However, many software developers believe that the compiler will find all defects. Unfortunately there are many typing error mistakes that are not found by the compiler. For example, in C, typing “=“instead of “= =“can cause an assignment instead of a comparison. Although compiler can find about 90% defects but what’s about the other 10%? Many people believe that the other 10% could be done by testing. However, many programs will run even when they have defects. In fact, they can have a lot of defects and still pass many tests. To find even a large percentage of the defects in a program, we would have to test almost all the logical paths and conditions. And to find all of the defects in even small programs, we would have to run an exhaustive test that could be expensive and require a lot of efforts.

Today most software developers spend a lot of time trying to get their software to work then spend more time fixing defects and reported problems. This is a major issue for the business but many software managers are not trained to deal with this. They only focus on the schedule, how to deliver within certain date but do not know much about the financial business. Of course senior managers know but they are too busy running company and not pay attention at project so the ultimate people who really care are customers. Of course the customers have choices and when they are not happy with the quality, they do business with somebody else.

I also found that most software developers are not trained in identify and fixing security defects. A security defect is any design error that allows hackers, criminals, or terrorists to obtain unauthorized access or use of a software system. Since many of these defects do not cause functional problems, or runtime errors they will pass all of their functional tests but the software has potential security vulnerabilities that could create problems in the future.

My view is to ensure quality, we must build quality into the way we work and that defects are serious issue that require significant attention from everybody. The best way to prevent defects is better training and training should start as early as possible in every university and should be emphasized in all computing classes.

Sources

  • Blogs of Prof. John Vu, Carnegie Mellon University