Software Testing
Testing is the execution of a program to find defects. A good test case is one that has a high probability of detecting defect. However, industry studies found that even with the best test case, developers can only catch about 70% of the defects and the other 30% still remain after the software is released to users. Because the limit of testing, it is recommended that developers remove as many defects as possible by other methods such as software reviews and inspections before testing phase to improve the quality of the software products.
There are many reasons for software to have defects. The most common is software requirements are not well defined and often change. These changes cause developers to change what they are doing and it creates defects. That is why Requirements Engineering (RE) is an important skill because it focuses on identify any ambiguous or missing information during the requirements phase. It forces developers to look at the entire requirements to determine “Testability attributes”. For every requirement, developers must be able to write a test case. If they cannot test it then either the requirement is ambiguous or developers do NOT understand the requirement. In either case, a discussion with users to clarify the requirement is necessary. That is why testing must be planned immediately after the requirements phase. However, lacking of the requirements engineering skill, many developers do NOT plan testing until they complete coding. This is too late and it is a big mistake. That is why most software have too many defects and too costly.
Another common mistake is the Test strategy is NOT well defined. Most developers focus on testing functions but NOT on integration or build. They believe as long as they test all requirements than everything will be good. They are NOT trained in “System Thinking” to understand the robustness of software by concentrate on critical portions of the system such as capacity and performance. Many times, all functions independently work well (Passing functions test) but when you put them together, the system does NOT work anymore or perform very slow (Failing integration test or Performance test). These types of defect are difficult and require a lot of time and efforts to fix.
Another common mistake is the Test resources are NOT adequate. Either project manager assigns too little time for testing or test people are not brought onto project until late. If the test group does NOT understand the system well than they cannot do good job in finding defects. Many project managers do NOT clearly distinct testing activity and developing activity. Since testing is often regarded as the last activity, if a project is late, test is often eliminated to meet the scheduled deadline. Of course, many software have defects and users are very angry.
Another common mistake is on the knowledge and skill of testers. Many testers do NOT make distinction between test plan (objectives, approach, resources required, etc.) and test procedure (specific inputs, the expected outputs, directions to the tester, etc.). Many would program test cases rather than review test plan for adequacy, test procedures for completeness, and failure mode effects analysis to identify probable error-causing inputs as test data. Even when they test, they do NOT perform coverage analysis identify which control paths have been exercised by the test data and which have not. Most do NOT assess test effectiveness to determine how well the test data have exposed defects contained in the software. Without adequate training, many do NOT consistently apply regression testing to ensure that program changes have not impacted correct system functioning. They do NOT know how error discovery rate can be analyzed to support a rational product release decision.
During my trip to several Asian countries last year. I found that many universities did NOT have courses on software testing. Testing was taught as part of programming language classes. Many students did NOT understand Unit testing, Integration testing, System testing, alpha testing, beta testing, Regression testing, and Acceptance testing. To them everything was just a “big test” to make sure that software executed and did NOT crash. Unless this view is changing and appropriate training is in place, software quality will NOT be improved.
Sources
- Blogs of Prof. John Vu, Carnegie Mellon University
