Articles and papers on testing from Software Engineering and DoD newletters, journals, theses and magazines, that are directly available on line. In some cases a membership or subscription is required to read the full text online.
10 Quality Assurance Traps - Ten common mistakes that organizations make when seeking Quality Assurance during the software development life-cycle.
Abstract:
Pairwise testing has become a popular approach to software quality assurance because it often provides effective error detection
at low cost. However, pairwise (2-way) coverage is not sufficient for assurance of mission-critical software. Combinatorial testing beyond pairwise is rarely used because good algorithms have not been available for complex combinations such as 3-way, 4-way, or more. In addition, significantly more tests are required for combinations beyond pairwise testing, and testers must determine expected results for each set of inputs. This article introduces new tools for automating the production of complete test cases covering up to 6-way combinations.
Climbing Mount Automation - This paper by Bob Johnston of Scriptech, discusses How and why to integrate manual and automated test projects. He compares different methodologies rangin from using Subject Matter Experts to Subject Matter Robots.
Experiences in Testing Pocket PC Applications - Presented by Ibrahim El-Far (Florida Institute of Technology), Florence Mottay (J.D. Edwards), & Dr. Herbert Thompson (System Integrity) at the Internet & Software Quality Week Europe Conference (QWE 2001) November 2001. Promising model-based approaches have emerged in the field of software testing during the past decade or so. In this presentation, the focus is on the authors' experiences with using finite state machines to test a host of applications created by Microsoft for the Pocket PC platform. Creating a state model of each of these applications facilitated the tasks of building test automation and verifying test results. The process followed is explained, the results presented, and some of the lessons shared.
Generating Test Data from State-Based Specifications - Written by Jeff Offutt, Shaoying Liu, Aynur Abdurazik and Paul Amma in the Journal of Software Testing, Verification & Reliability V. 13, No. 1, March 2003. ABSTRACT: This paper presents general criteria for generating test inputs from state-based specifications. The criteria include techniques for generating tests at several levels of abstraction for specifications (transition predicates, transitions, pairs of transitions and sequences of transitions). These techniques provide coverage criteria that are based on the specifications, and are made up of several parts, including test prefixes that contain inputs necessary to put the software into the appropriate state for the test values. The test generation process includes several steps for transforming specifications to tests. These criteria have been applied to a case study to compare their ability to detect seeded faults.
The big new force that is propelling the good enough idea is the explosion of market-driven software. With a passion roughly proportional to the price of Microsoft stock, companies are looking for the shortest path to better software, faster, and cheaper. They are willing to take risks, and they have little patience for the traditional moralistic arguments in favor of so-called good practices. Much of the traditional lore of software project management seems irrelevant or stilted when applied to market-driven projects. It's time that we developed approaches and methodologies that apply to the whole craft, not just to space missions, medical devices, or academic experiments. Good enough is a model that encompasses high-reliability products as well as high-entertainment products. Whether you call the idea good enough, or choose another buzzword like economical, pragmatic, or utilitarian, the basic idea remains the same: our behavior should be guided by reason, not compulsion. Beyond the notion of best practices is a more fundamental idea: best thinking. As the good enough idea continues to emerge, the quality of one's thinking, rather than conformance to formalities, will become the issue. Formalities, and the authority behind them, will be re-examined. No wonder so many authorities consider good enough to be a dangerous idea
"IPL delivers software development, systems integration and consultancy services to customers who demand the highest standards of quality and dependability. We also develop leading edge testing tools that are used by software developers around the world.
Founded in 1979 and based in our wholly owned premises in the World Heritage City of Bath, UK, IPL now employs around 240 staff."
National Aeronautics and Space Agency (NASA) -- A Software Quality
Model and Metrics for Identifying Project Risks and Assessing
Software Quality - This paper explains a Software Quality Model and then uses it as a basis for discussions on quality attributes and risks. Risks that can be determined by a metrics program are identified and classified. The software product quality attributes are defined and related to the risks. Specific quality attributes are selected based on their importance to the project and their ability to be quantified. The risks and quality attributes are used to derive a core set of metrics relating to the development process and the products, such as requirement and design documents, code and test plans. Measurements for each metric are defined and their usability and applicability discussed.
Quality Cost Analysis: Benefits and Risks - by Cem Kaner (1996) - Quality costs are the costs associated with preventing, finding, and correcting defective work. These costs are huge, running at 20% - 40% of sales. Many of these costs can be significantly reduced or completely avoided. One of the key functions of a Quality Engineer is the reduction of the total cost of quality associated with a product. The author presents six useful definitions, as applied to software products.
Software Testing Principles - This page presents an overview of software testing which includes the following topics: testing basics, functional testing features, manual tools, automated programmer tools, automated non-programmer tools, and testing the multi-platform system. A bibliography of books is included.
Software-Reliability-Engineered Testing - Crosstalk (June 1996) - An article by John D. Musa, AT&T Bell Laboratories and James Widmaier,
National Security Agency in the june, 1996 issue of CrossTalk (a publication of the DoD Software Technology Support Center (STSC))
Software reliability engineered testing (SRET) is system testing designed and guided by quantitative software system reliability objectives and expected field usage and criticality of different operations. It can be applied to both custom software in development and off-the-shelf software of any singular or collectively integrated size. Ada and other object-oriented (OO) languages that tout reuse should now take advantage of this concept for their quantitative reliability metric.
Test Selection Based on Finite State Models - Authors: Fujiwara, S. v. Bochmann, G. Khendek, F. Amalou, M. Ghedamsi, A. This paper appears in: Software Engineering, IEEE Transactions on Software Engineering , Volume: 17, Issue: 6. ABSTRACT: A method for the selection of appropriate test case, an important issue for conformance testing of protocol implementations as well as software engineering, is presented. Called the partial W-method, it is shown to have general applicability, full fault-detection power, and yields shorter test suites than the W-method.
The Cost of Testing Software - Presented by Robert Vienneau, (DACS) at the Annual Reliability and Maintainability Symposium Proceedings. Orlando, Florida, 1991. IEEE Catalog Number: 91CH2966- 0
ABSTRACT: A simple quantitative cost model that allows a manager to determine the optimum amount of time needed to test a software system has been developed. At the minimum cost point, the cost of an infinitesimal increment of testing equals the marginal benefit to be gained from the increment. The model can also be used to estimate the cost of testing software if the release data is determined by some other criteria, such as reliability. Since the cost of testing software depends on the pattern of failures, the model is based on underlying software reliability models. The cost model is explicitly developed so as to apply to a wide class of reliability models. It is noted that special cases result from choosing specific models, for example, the Goel-Okumoto nonhomogeneous Poisson process model, in which all bugs have an equal impact on the failure rate, and the Musa-Okumoto logarithmic Poisson process model, in which bugs found earlier have a larger impact.
The Number of Tests Required for Branch Coverage - Crosstalk (September 1996) - This article addresses the number of test cases needed to achieve branch coverage and introduces a new bound called beta branch , which is calculated upon the program control flow structure. - Crosstalk is a publication of the DoD Sofware Technology Support Center (STSC).
DACS Gold Practice Initiative
