ROI Dashboard FAQ
1.0 What is the DACS ROI Dashboard?
2.0 What software technologies are covered by the Dashboard?
3.0 What variables does the Dashboard show the impact of software technology on?
4.0 How does the Dashboard show the impact of software technology on the selected variables?
5.0 How do I use the Dashboard?
6.0 Where does the data displayed on the Dashboard come from?
References
1.0 What is the DACS ROI Dashboard?
The Data & Analysis Center for Software (DACS) Return On Investment (ROI) Dashboard provides the user with information on the costs and benefits of various improvements to software technology. This data was obtained from open source literature. The user obtains access to information in the database through a Web interface.
2.0 What software technologies are covered by the Dashboard?
The following software technologies are covered by the Dashboard:
- Capability Maturity Model (CMM) Software Process Improvement
- Capability Maturity Model - Integrated (CMMI) Software Process Improvement
- Cleanroom software engineering
- Extreme Programming
- Goal Question Metric
- ISO 9001
- Inspections
- The Personal Software Process (PSP)
- Software reuse
- Software Process Improvement (SPI)
- The Team Software Process (TSP)
2.1 What is CMM Software Process Improvement?
The Capability Maturity Model (CMM) for software has been used by many organizations to identify best practices useful in helping them increase the maturity of their processes and improve long-term business performance. The Software Engineering Institute also supports the Capability Maturity Model - Integration and CMMs for people and teams. See Paulk et al. (1995) for a definition of the CMM, including Key Practice Areas (KPAs) for each of the five maturity levels.
2.2 What is CMMI Software Process Improvement?
After the development of the CMM, the Software Engineering Institute developed a variety of maturity models in different areas. The Capability Maturity Model Integrated (CMMI) supercedes and integrates some of these models. Specifically, the most comprehensive version of the CMMI, in either the staged or continuous form, covers
- Systems Engineering
- Software Engineering
- Integrated Product and Process Development
- Supplier Sourcing.
2.3 What is Cleanroom?
Cleanroom software engineering is intended to achieve or approach zero defects with certified reliability. The Cleanroom methodology provides a complete discipline for planning, specifying, designing, verifying, coding, testing, and certifying software. In a Cleanroom development, correctness verification replaces unit testing and debugging. After coding is complete, the software immediately enters system test with no debugging. All test errors are recorded from the first execution of the program, and no private testing is allowed. As opposed to many development processes, the role of system testing is not to test in quality; the role of system testing is certify the quality of the software with respect to the system specification. Cleanroom software engineering draws on incremental development processes, top down design, function theory, box structure specifications and design, inspections, usage specification and testing, and software reliability modeling.
2.4 What is Extreme Programming?
EXtreme Programming (XP), as described by Beck for example, is the most popular form of Agile Programming. The principles of agile programming, as put forth in the Agile Manifesto, are to value:
- Individuals and interactions over processes and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a plan
2.5 What is Goal Question Metric Software Process Improvement?
The Goal-Question-Metric paradigm is an approach to software measurement invented by Victor Basili. Overall goals are used to develop questions about software processes and products. These questions in turn drive the identification of metrics to collect and their analysis.
2.6 What is ISO 9001?
The ISO 9000 family of standards specify quality management systems. ISO 9001 Quality Management Systems - Requirements (2000) is now the only standard in the family against whose requirements a quality system can be certified by an external accredited agency. The standard recognizes that "product" applies to services, processed material, hardware and software produced for, or required by, a customer.
ISO 9001:2000 specifies requirements for a quality management system for an organization that:
- Demonstrates its ability to consistently meet customer requirements
- Demonstrates its ability to consistently meet applicable regulatory requirements
- Aims to enhance customer satisfaction through effective application of the system
- Achieves continual improvement of its performance in pursuit of these objectives.
All requirements of this international standard are generic and are intended to be applicable to all organizations, regardless of type, size, and product provided.
2.7 What are Inspections?
Software Inspection is the practice, originated by Fagan, of conducting a visual examination of software product following a well-defined and disciplined process. Inspections typically have defined roles (e.g., moderator, reader, author, inspector, and recorder) and defined phases (e.g., planning, overview, preparation, inspection meeting, rework, and follow-up.) The purpose of an inspection is usually to detect and identify defects.
2.8 What is the Personal Software Process?
The Personal Software Process (PSP) helps engineers
- Manage the quality of their projects
- Make commitments they can meet
- Improve estimating and planning
- Reduce defects in their products
Based on practices in the CMM, the PSP can be used by engineers as a guide to a disciplined and structured approach to software development. The PSP is a prerequisite for the TSP.
2.9 What is Reuse?
Software reuse is the use of an existing software artifact in a new context, either in the system for which it was initially developed or in another system. Reuse includes both compositional approaches, in which a system includes reused parts from, for example, a reuse library system, and generative approaches, in which a system with a parametrized standard architecture is automatically produced. Generative approaches are typically domain-specific.
2.10 What is Software Process Improvement?
Software Process Improvement (SPI) is a defined process to increase the capability and maturity of an organization’s software processes. Typical organizations set up a Software Engineering Process Group (SEPG) to supervise process improvement efforts. An SEPG will use a model, such as the Capability Maturity Model - Integration (CMMI) to assess the organization’s processes at various points of an improvement effort.
2.11 What is the Team Software Process?
The Team Software Process (TSP), along with the Personal Software Process, was developed by the Software Engineering Institute to help the high-performance engineer:
- Ensure quality software products
- Create secure software products
- Improve process management in an organization.
Engineering groups use the TSP to apply integrated team concepts to the development of software-intensive systems. After a four-day launch process, the TSP provides a defined process framework for managing, tracking and reporting the team's progress. TSP helps an organization establish a mature and disciplined engineering practice that produces secure, reliable software.
3.0 What variables does the Dashboard show the impact of software technology on?
The Dashboard provides the user with capabilities to examine the empirical distribution of the impact of software technology improvements on Productivity, Quality, Return On Investment, Rework, Project Cost, Cost of the Improvement, Cycle Time and Schedule Variance.
3.1 What is Productivity?
The amount of output produced for unit input. Software productivity is typically assessed in terms of Source Lines Of Code (SLOC) per staff-day or Function Points per person-month. The Dashboard graphs the percent improvement in productivity.
3.2 What is Quality?
The definition of quality is a widely discussed topic. The Dashboard displays the impact of software technologies on two measures of quality:
- The percent decrease in defect density. Defect density is measured in units such as delivered defects per Thousand Source Lines Of Code (KSLOC) or defects found during acceptance testing per thousand Function Points.
- Percent of defects found. These can be the percent of defects found by an activity, such as Software Quality Assurance or Inspections, or the percent of total defects found before delivery.
3.3 What is ROI?
Return On Investment (ROI) is the ratio of net benefits to the cost of the improvement. Benefits are net of the cost of the investment; that is, net benefits are the difference between total benefits and the investment that was made to achieve them. Both are measured in the same units, for example, dollars or staff hours. Typically, benefits are savings obtained by reduced rework.
3.4 What is Rework?
Rework is the cost (often measured in terms of time or dollars) required to alter a component that has already been completed as a result of a defect. The Dashboard displays the change in rework measured as a percentage of overall project cost.
3.5 What is Project Cost?
Project cost is the total amount budgeted for the project.
3.6 What is the Cost of the Improvement?
The Cost of the Improvement is the total cost required to implement the improvement. Usually this includes the time required to train the staff and/or perform the new process (as in Formal Inspections).
3.7 What is Cycle Time?
The time from inception to product release; total schedule length for a software development project.
3.8 What is Schedule Variance?
The difference between the amount of work performed and all scheduled work at the scheduled completion date. This definition comes from Earned Value Analysis. The amount of work performed is measured by the Budgeted Cost of the Work Performed (BCWP). The work scheduled is measured by the Budgeted Cost of the Work Scheduled (BCWS). Schedule Variance is negative for a project that runs over schedule and positive for a work completed ahead of schedule.
4.0 How does the Dashboard show the impact of software technology on the selected variables?
The Dashboard displays empirical distributions of certain variables for organizations adopting certain user-specified software technologies. These displays graph distribution-free statistics (medians and approximations of first and third quartiles) and parametric statistics for the distributions (means and standard deviations). The distribution-free statistics are shown on box plots, while the parametric statistics are shown on bar graphs. The Dashboard also presents timelines showing when organizations implemented the technologies. Detailed information is presented on each entry in the database. This detailed information allows the user to see how the impact on a variable was presented in the source document, background information and other observations about the organization, and the reference from which the data was obtained.
A boxplot, also called a box-and-whiskers plot, displays the empirical distribution of a single variable. Half the distribution is in the center box. Whiskers, at the top and bottom of the box, show the extent of most of the remainder of the distribution. Finally, outliers and extreme values are plotted beyond the whiskers. Tukey (1977) first described boxplots. The boxplot shows various statistics. Boxplots on the DACS ROI Dashboard display two measures of central tendency, the mean and the median. Traditional boxplots do not display the mean. The lower and upper edges of the box, known as “hinges”, approximate the first and third quartiles of the distribution.
A bar graph displays the empirical distribution of a single variable. The central tendency of the data is shown by the mean. The dispersion of the data is shown by dotted lines a standard deviation below and above the mean. The extent of the graph shows the range of the data.
4.3 What is the 'text' display option?
The 'text' display option lists statistical values for each selected improvement and variable. The statistical values shown are total number of data points, minimum, maximum, median, mean, standard deviation, 25th percentile and 75th percentile.
A measure of central tendency. The median is such that half the sample is less than its value. The median and the mean are equal for Gaussian (bell-shaped) distributions. The median is less sensitive to outliers and extreme values. Medians are discussed and defined in many introductory statistics textbooks.
A measure of central tendency. The mean is the quotient of the sum of the values and the number of elements of a sample. Means are discussed and defined in many introductory statistics textbooks.
The first quartile is such that a quarter of the sample data points are less than its value. The second quartile is more commonly called the median. The third quartile is such that three quarters of the sample data points are less than its value. The distance between the third and first quartile is a measure of the dispersion of the data. Percentiles, such as quartiles, are discussed and defined in many introductory statistics textbooks.
An approximation to a quartile. Hinges were defined by Tukey to simplify calculations of the limits of the boxes in box-and-whiskers plots. The lower hinge is the median of the half of the sample data points less than or equal to the median. The upper hinge is the median of the half of the sample data points greater than or equal to the median. These hinges differ from the first and third quartiles in how interpolation is handled when a quartile or hinge lies between two of the sample data points.
In the context of box-and-whiskers plots, an outlier is a data point that is
In the context of box-and-whiskers plots, an extreme value is a data point that is either three box lengths above the upper edge of the box or three box lengths below the lower edge of the box.
A measure of dispersion. The square of the standard deviation is the quotient of a certain sum and one less than the number of elements of a sample. That sum is the sum of the squares of the difference between the value of each element of the sample and the mean. The standard deviation is in the same units of measure as the elements of the sample.
For Gaussian (bell-shaped) distributions, 68.3% of the population is within one standard deviation of the mean, 95.4% is within two standard deviations, and 99.7% is within three standard deviations. Standard deviations are discussed and defined in many introductory statistics textbooks.
See the user’s manual.
Mostly from the open literature. Some of the data is from articles in Communications of the ACM, CrossTalk, the IBM Systems Journal, IEEE Transactions on Software Engineering, Computer, Software, the Journal of Systems and Software, Software Practice and Experience, and Software Process – Improvement and Practice. Some of the data is from papers presented at conferences, such as conferences sponsored by the Software Engineering Institute and Software Engineering Process Group (SEPG) conferences. Some of the data is from technical reports.
Some of the data has been collected by the DACS through surveys.
-
K. Beck, “Embrace Change with Extreme Programming”, IEEE Computer. p. 70-77, October 1999.
-
D. N. Card, “Learning from our Mistakes with Defect Causal Analysis”, IEEE Software. p. 56-63, January-February 1998.
-
CMMI Product Team, CMMI for Systems Engineering, Software Engineering, Integrated Product and Process Development, And Supplier Sourcing (CMMI-SE/SW/IPPD/SS, V1.1) Continuous Representation. CMU/SEI-2002-TR-011, March 2002.
-
CMMI Product Team, CMMI for Systems Engineering, Software Engineering, Integrated Product and Process Development, And Supplier Sourcing (CMMI-SE/SW/IPPD/SS, V1.1) Staged Representation. CMU/SEI-2002-TR-012, March 2002.
-
W. J. Dixon and F. J. Massey, Jr., Introduction to Statistical Analysis (Fourth edition). McGraw Hill, 1983.
-
Mark C. Paulk et al., The Capability Maturity Model: Guidelines for Improving the Software Process. Addison Wesley, 1995.
- J. W. Tukey, Exploratory Data Analysis. Addison Wesley, 1977.
Copyright 2010 by ITT Industries
|
DACS Gold Practice Initiative
ROI Dashboard
