Software Testing Best Practices
27 May. 2011 Software Testing & QA
Economics of software testing is to determine and predict the defects early by using predictable models and implying strategies and different test methodologies to identify those at early stages.
Psychology of software testing is to destructively test the application by identifying as many exceptional or out of the box scenarios or sometimes called as the third vision.
Software Testing is a systematic activity but it also involves economics and human psychology. – Glenford J. Myers
A set of good test scenarios evaluates every possible permutation and combination of a program during ideal conditions. In addition, a software Test Engineer needs the proper vision to successfully test a piece/whole application to comply with the standards and quality expected by the industry. Whenever Software Test Engineers tests a program, they should add some value in it rather than performing only the requirements conformance and validation. A systematic and well planned test process adds the value of quality and reliability to the software program.
The most important considerations that come with offering software testing services are the issues of psychology, leading to the set of principles and guidelines to evaluate the software’s quality.
Ten Commandments of Software Testing Best Practices are
1. Software Testing is the process of exercising a software component using a selected set of test cases, with the intent of
- revealing defects
- evaluating quality
A Software Test Engineer executes the software using test cases to evaluate properties such as reliability, usability, maintainability, and level of performance. Test results are used to compare the actual properties of the software to those specified in the requirements document as quality goals. Deviations or failure to achieve quality goals must be addressed.
Software Testing has a broader scope rather being only limited to the execution of program or detecting errors more rigorously as described in the test process improvement models such as TMMi framework models.
2. When the test objective is to detect defects, then a good test case is one that has a high probability of revealing a yet undetected defect(s).
Principle 2 supports a strong and robust designing of test cases. This means that each test case should have a goal to identify or detect a specific type of defect. The Software Test Engineer approaches the scientific method of designing the tests to prove or disapprove the hypothesis by the means of test procedures.
3. A test case must contain the expected output or result.
Principle 3 supports the fact that a test case without expected result is of zero value. Expected output or result determines whether a defect has been revealed or the conditions have been passed during the execution cycle.
4. Test cases should be developed for both valid and invalid input conditions.
Use of test cases that are based on invalid inputs is very useful for revealing defects since they may exercise the code in unexpected ways and identify unexpected software behavior. Invalid inputs also help developers and Software Test Engineers to evaluate the robustness of the software, that is, its ability to recover when unexpected events occur (in this case an erroneous input).
For example, software users may have misunderstandings, or lack information about the nature of the inputs. They often make typographical errors even when complete/correct information is available. Devices or software program may also provide invalid inputs due to erroneous conditions and malfunctions.
5. The probability of the existence of additional defects in a software component is proportional to the number of defects already detected in that component.
Principle 5 supports to the fact that the higher the number of defects already detected in a component, the more likely it is to have additional defects when it undergoes further testing.
For example, if there are two components A and B, and Software Test Engineers have found 20 defects in A and 3 defects in B, then the probability of the existence of additional defects in A is higher than B. This empirical observation may be due to several causes and degree of influence of the software factors. Defects often occur in clusters and often in code that has a high degree of complexity and is poorly designed.
6. Test Cases must be repeatable and reusable.
Principle 6 is utmost important and plays a vital role supporting the fact that it is also useful for tests that need to be repeated after defect repair. The repetition and reuse of tests is also necessary during regression test (the retesting of software that has been modified) in the case of a new release of the software.
7. Testing should be carried out by a group that is independent of the development group.
This principle holds true for psychological as well as practical reasons and does not say that it is impossible for a programming organization to find some of its errors, because organizations do accomplish this with some degree of success. Rather, it implies that it is more economical for testing to be performed by an objective, independent party which gives a direction of the third vision by the means of test cases. Finally, independence of the testing group does not call for an adversarial relationship between developers and Software Testing Engineers.
8. Test Activities should be well planned.
Test plans should be developed for each level of testing, and objectives for each level should be described in the associated plan. The objectives should be stated as quantitatively as possible. Plans, with their precisely specified objectives, are necessary to ensure that adequate time and resources are allocated for testing tasks, and that testing can be monitored and managed. Test planning must be coordinated with project planning. A test plan is a roadmap for testing which should be mapped to organizational goals and policies pertaining to the software program. Test risks must be evaluated at each level of testing.
Careful test planning avoids wasteful “throwaway” tests and unproductive and unplanned “test–patch–retest” cycles that often lead to poor-quality software and the inability to deliver software on time and within budget.
9. Avoid throwaway test cases unless the program is truly a throwaway program.
Whenever the program has to be tested, the test cases must be reinvented. More often than not, since this reinvention requires a considerable amount of work, Software Testing Engineers tend to avoid it. Therefore, the retest of the program is rarely as rigorous as the original test, meaning that if the modification causes a previously functional part of the program to fail, this error often goes undetected.
10. Test results should be inspected meticulously.
This is probably the most obvious principle, but again it is something that is often overlooked. We’ve seen numerous tests that show many subjects failed to detect certain errors, even when symptoms of those errors were clearly observable on the output listings. Put another way, errors that are found on later tests are often missed in the results from earlier tests.
- A failure may be overlooked, and the test may be granted a “PASS” status when in reality the software has failed the test. Software Testing may continue based on erroneous test results. The defect may be revealed at some later stage of testing, but in that case it may be more costly and difficult to locate and repair.
- A failure may be suspected when in reality none exists. In this case the test may be granted a “FAIL” status. Much time and effort may be spent on trying to find the defect that does not exist. A careful re-examination of the test results could finally indicate that no failure has occurred.
Software Testing is an extremely creative and intellectually challenging task.
Creativity required in testing a large program exceeds the creativity required in designing or developing that program. We already have seen that it is impossible to test a program sufficiently to guarantee the absence of all errors. This requires a systematic test process and methodologies to design the robust test cases.
This principle supports that a Software Testing Engineer needs to have
- Comprehensive knowledge of the software engineering discipline.
- Knowledge from both experience and education as to how software is specified, designed, and developed.
- Knowledge of fault types and where faults of a certain type might occur in code constructs.
- Reasoning like a scientist and propose hypotheses that relate to presence of specific types of defects.
- A good grasp of the problem domain of the software that he/she is testing.
- Create and document test cases. To design the test cases the Software Testing Engineer must select inputs often from a very wide domain.
- Design and record test procedures for running the tests.
- Execute the tests and is responsible for recording results.
- Analyze test results and decide on success or failure for a test. This involves understanding and keeping track of an enormous amount of detailed information.
- Know the method for collecting and analyzing test-related measurements.
Learn how the software testing best practices can help your application reach a level of quality aspired by many in the industry. Come talk to us on 1-877-RISHABH (1-877-747-4224) or contact us here to learn how we can help you achieve this through our application testing services.