Introduction
Chief
scientist, Caper Jones, of Software Productivity Research in Massachusetts,
estimates that 60 percent of the United State’s software work force is
dedicated to fixing software errors that could have been avoided. In addition,
he states there are only 47 days in a calendar year dedicated to doing actual
development or enhancement of software applications. Most of the software engineering
work is wasted due to fixing errors or working on projects that are eventually
cancelled (Chapman, 2000).
It
appears product reliability has fallen to the waste side. The 1980s emphasized
quality control, however by the 1990s innovation at top speed has taken top
priority. In America, being first to market is the ultimate goal. Forget the
errors as those can be fixed later by technical support staffs or customer
service staffs (Barron, 2000). However, there is a new era of computing upon
us.
The current era of desktop
computing will soon pass evolving to a new era of globalization. Software
globalization will form the new infrastructure of our society. It will require
desktop computers to be an essential business tool rather than the occasional productivity
enhancer. In essence, this new era will require high dependability and
reliability due to software computing now performing life critical functions
(Morris, 2000). Therefore, there must be a better method to software
development without creating numerous defects.
One
approach to developing quality software with minimal defects is Cleanroom
Software Engineering (CSE). This academic research paper provides an in-depth
review of CSE including a description of what it is, the advantages and disadvantages
of using it, and real world examples of its use. Lastly, the paper concludes
with some thoughtful points on how CSE can be used to develop better
commercial-off-the-shelf (COTS) software.
Cleanroom Software Engineering (CSE) Defined
Harlan
Mills and his colleagues from IBM developed the CSE methodology in the early
1980s. They were part of IBM’s Federal Defense System where software failures
could mean millions of dollars and most importantly, human lives (Head, 1994).
This software methodology follows the same analogy as cleanroom fabrication of
semiconductors. Instead of trying to clean dirt off the semiconductor wafers
after production, the object is to prevent the dirt from getting into the
production environment in the first place. The reason for this is that defect
prevention is more cost effective than defect removal (Deck, 1994). Therefore,
in software development, the CSE methodology eliminates or avoids as many
defects as possible before software execution using controlled and measurable statistics
(Stavely, 1999).
CSE
focuses on theory-based engineering practices of software development and
certification while using the Capability Maturity Model for Software (CMM) as
its framework (Oshana, 1998). This framework (i.e. CMM) describes the key
elements of an effective software process such as planning, engineering,
managing development and maintenance. In addition, CMM establishes a means to
measure an organization’s software development process by maturity levels
(Oshana & Linger, 1999). Nevertheless, the main point is that CSE
encompasses CMM as part of its software development process.
CSE
spans the entire software life cycle. It has an 80-20 life cycle. This means 80
percent of the time is dedicated to design while 20 percent is writing code. In
contrast, the traditional life cycle is 40-20-40 where 40 percent is design, 20
percent is coding, and 40 percent is unit testing. There is absolutely no unit
testing as part of the CSE process (Head, 1994). The untested software product
is expected to work. If it does not, the defects are analyzed to determine how
the software development process should be improved. Then the defective product
is discarded. According to Harlan Mills, the most important tool to CSE is the
wastebasket (Head, 1994).
CSE is based on two main
principles. They are the design and testing principles. The design principle
consists of mathematical functional verification of the product. Here reviewers
of the software product use mathematical functions to argue the correctness of
the original design. The testing principle focuses on certifying the
reliability of software product (i.e. measuring the quality). This technique
uses statistical testing based on expected system usage rather than trying to
remove defects (Becker, Deck, & Janzon, 1996).
These may be the two
fundamental principles to CSE. However, there are four main components to the
CSE framework. These components are discussed next.
Main
Components
The
four main components to CSE include the following: incremental development
process model; formal methods for specification and design; correctness
verification of developed code; and statistically based, independent testing
(Grillo & Tewari, 1997). Figure one in Appendix A shows the CSE Model. Each
of these components is necessary to develop software code that prevents errors
from the beginning of development.
Step 1:
Incremental Development Process Model.
In
the incremental development process of CSE, each increment is a functional
subset of the final software product. These increments are developed and tested
in the environment similar to that of the final software application, so
inferences about field quality can be drawn from the test results (Deck &
Whittaker, 1997). If the increment adheres to pre-established quality
standards, then the next incremental development process is started. If not,
then development is stopped and the process is fixed (Oshana, 1998).
By having many increments
(i.e. mini releases) within one software project, the results from one
increment can be used to modify the process for subsequent increments. This is
a great advantage to any iterative process (i.e. incremental, spiral,
evolutionary) because it has the ability to use feedback from early iterations
to improve the process. Thus, the chances for success are greater. Therefore,
the incremental development process is a technique for managing risk and
incorporating risk alleviation into the CSE process (Deck & Whittaker,
1997).
Step 2:
Formal Methods for Specification and Design.
The
Box Structure Method is typically used for specification and design of the CSE
process. It is used to bridge the gap between system-level thinking and
design-level thinking (Deck, 1996, October 10-11). The key concept of CSE is
that a software program is a rule for a mathematical function. Thus, a software
program executes transformations based on inputs (i.e. domains) to outputs
(i.e. ranges). The transformations are specified as functional mappings (Grillo
& Tewari, 1997).
The box structure method is
used as a stepwise refinement and verification process. It defines three levels
of abstractions for defining software programs as rules for functions. The
three levels of abstractions are the behavioral view, the finite state machine
view, and the procedural view (Oshana, 1998). These levels of abstraction are
classified as black, state, and clear boxes respectively. The black box defines
usage hierarchy and may be referenced by the state and clear box. The state box
view details about the data implementation while the clear shows the procedural
view (Deck, 1996, October 29-30). Nevertheless, the box structure method helps
to create clear and well-defined specifications.
Step 3:
Correctness Verification of Developed Code.
Correctness verification is
believed to be the heart of CSE. This component is primarily responsible for
the dramatic improvements in quality (Head, 1994). Here the testing process is
completely separated from the development process. Team reviews use
mathematical verification techniques to verify the correctness of the software
program. Thus, by being able to reason mathematically the effect of each code
statement, this proves that the software program can meet its required
specifications (Grillo & Tewari, 1997).
This step compares to unit
testing of traditional software development methods. Unit testing is where the
program developer will test their program for the purpose of finding defects.
However, in CSE, team reviews replace this. There are two main reasons why team
reviews are considered more effective than unit testing. They are more thorough
and cost effective (Deck, 1997, May 27-30).
Step 4:
Statistically Based, Independent Testing.
Lastly,
the software program is tested based on statistical principles. First all
possible software uses are generated into statistical subsets. Performance is
measured on these subsets based on the usage model developed. Then the expected
operational use is represented in a usage model. Lastly, test cases are
randomly generated and executed in the operational environment. Any failures
are interpreted according to statistical models (Oshana, 1998).
An
independent team does the statistical testing. This testing is viewed as a
sampling problem in CSE versus coverage testing for debugging in the traditional
approach. The team does not assist the development group in getting the
software product to an acceptable level of quality, but rather certifies its
reliability (Grillo & Tewari, 1997).
Summary
In
summary, CSE is an error-removal software development methodology. It centers
around two principles – design and testing. In addition, it has four main
components to its process. They are the incremental development process model,
formal methods for specification and design, correctness verification of
developed code, and statistically based, independent testing.
These components
differentiate it from most traditional software development models. There are
three areas of differentiation. First, the programmer is removed from any
testing activity. Second, the programmer must do formal (measurable)
verification. Lastly, all testing must be statistical in nature (Glass, 1999).
Thus, it would appear CSE has a distinct advantage over all other software
development models. However, that may not be the case. Next, a review of the
advantages and disadvantages is presented.
Advantages & Disadvantages of CSE
CSE can be difficult to understand. This is
especially true due to the unavailability of literature on CSE. However,
publications are forthcoming as more organizations embrace it (Deck, 1997, May
27-30). In this section, the advantages and disadvantages of CSE are discussed.
Advantages
CSE has at least three main
advantages. They are improved quality, increased productivity, and improved
software maintainability. CSE improves quality by preventing errors in the
first place. Thus, there is a reduction in the overall cost of the program
(Kelly & Oshana, 1996). For example, from 1987 to 1994, software code
developed with CSE only had 2.3 defects per thousand lines of code measured
from the first time of execution. In contrast, industry averages using
traditional software development methods were estimated at 10 defects per
thousand lines of code after unit testing (Stavely, 1999).
Another advantage CSE has over
traditional software development methods is productivity. Productivity is
increased due to the reduced time required to debug and rework software (Kelly
& Oshana, 1996). For example, the industry productivity rate for
non-comment source statements (NCSS) per engineer month is 120. Conversely,
with a trained cleanroom team, the productivity rate is 800 NCSS (Head, 1994).
That is an additional 680 NCSS just using CSE!
Lastly, software developed with CSE
has clear, well-defined specifications with a less-complicated design allowing
for easier maintenance (Kelly & Oshana, 1996). This is because CSE
incorporates team reviews as part of the process and uses the box structure
method for creating specifications. In a team review situation, there is not the
crutch of relying on unit testing to find defects. In addition, team reviews
generally are a more cost-effective means of improving quality and
disseminating knowledge (Deck, 1997, May 27-30).
In addition, the use of box structures creates a
less-complicated design. Box structures permit a user-view black box
specification that is written in terms of system usage. Therefore,
specifications are detailed, clear, and well-defined creating a comprehensible
view of the software program. This results in the software program being much
easier to maintain (Deck, 1994).
Despite these advantages, CSE is still not a widely
accepted software development practice (Deck, 1997, May 27-30). In fact, there
are quite of few critics. Next is a discussion of the disadvantages of CSE.
Disadvantages
There are three reasons why critics
dislike CSE. First, there is the belief that CSE is too theoretical, too
mathematical, and just plain too radical for software development. Even the
correctness proof portion of CSE can be difficult and time consuming especially
for non-safety-critical programs (Deck, 1997, May 27-30).
CSE does require incremental development, the use of
box structures for well-defined specifications, and statistical testing for
verification. CSE use requires intensive training. Once trained, an
organization can save time and money. Nevertheless, an organization can try to
add small parts of the CSE process to their normal software development method.
Once confidence is built, additional components of CSE can be added to their
original process (Henderson, 1995).
Second, unit testing is not part of
the CSE process. Denying the programmer access to the compiler is unrealistic
and can be counterproductive. This is especially true if the programmer does
not even know the language or environment, when the software program is started
(Deck, 1997, May 27-30). However, unit testing only builds confidence about the
specific paths tested. Thus, only high-frequency errors are found from the
user’s point of view. Unit testing is unlikely to find rare execution errors.
Conversely, CSE uses statistical testing which tends to find errors in the
order of seriousness and can uncover failures more effectively than randomly
through unit testing (Henderson, 1995).
Lastly, there is the belief that the
software industry is still in its infancy to incorporate such a rigorous
process as CSE. The software development paradigm must change from one of art
to one of science. In many engineering disciplines, failures are not an option
and definitely are not considered the norm. In order to implement CSE
successfully, one must not only have formal training but commitment from
management (Henderson, 1995). Next, a review of some organizations that have
successfully implemented CSE is discussed.
Real World Examples of CSE
Although CSE is not widely accepted,
it has been used in many industries. Industries such as aerospace,
telecommunications, graphical, and CASE tools have implemented CSE successfully
(Deck, 1997, May 27-30). Listed below is a sample of projects that have
successfully used CSE.
The first application of CSE was to develop the COBOL Structuring Facility (COBOL/SF) product at IBM. There were 80 thousand lines of code (KLOC) of which 52,000 lines were modified or added. This project was split into five increments of four to 19 KLOC. It was estimated that 1,500 to 3,000 errors were eliminated through team reviews. Productivity was measured to be 740 lines of code per labor month. Typically, the industry averaged only 150 lines of code per labor month (Deck, 1994). In addition, this product had required only a small fraction of its maintenance budget in its operating history (Foreman, 1997).
Another successful CSE implementation was with the Software Engineering Laboratory (SEL) of the NASA Goddard Space Flight Center. CSE was applied to a 40 KLOC project. Results of this project included a 69 percent increase in productivity, a 45 percent decrease in error rate, and approximately 60-80 percent decrease in resource usage compared to other SEL projects (Deck, 1994).
The US Department of Defense (DoD) Software Technology at the Picatinny Arsenal used CSE for software enhancements for a set of Army weapon systems (Deck, 1994). The program was approximately 90 KLOC. This project had results of a 4.2:1 productivity increase and a 20:1 return on investment (ROI) (Foreman, 1997).
One of the largest implementations of CSE was in the telecommunications industry. Ericsson used CSE to develop its OS32 operating system for telecommunication switches. This project had 350 KLOC. Ericsson had seen the following results: a 114 percent productivity increase in testing, a 70 percent increase in design productivity, a 157 percent increase in quality, a 50 percent decrease in failures per KLOC, and a 4.4 percent decrease in resource usage (Deck, 1994).
Results
like those above cannot be ignored. If history were any indicator of future
performance, it would appear that many organizations would benefit from
implementing CSE into its normal software development process. A prime example
of an industry who could substantial benefit from CSE is those that provide
commercial-off-the-shelf (COTS) software.
Thoughtful Points about COTS Software and CSE
It is a myth that software
must contain defects (Head, 1994). However, we tend to accept them as part of
the normal development process. This self-fulfilling prophecy prevents
defect-free software from being introduced to the marketplace (Head, 1994). For
example, when the author of this paper thinks of the next operating system from
Microsoft, she expects that there will be many errors.
Typically, COTS software
relies on updates and patches for errors to be fixed. It is readily apparent
that COTS software companies would benefit by incorporating CSE as part of
their software development process.
Although
this industry has heard about CSE, they believe it is not well-suited to the
COTS software environment. This is because developers must use new, unstable,
or poorly documented languages or platforms. In addition, most COTS software is
rarely used for life-critical applications (Deck, 1996, October 29-30).
However, as stated before, there is a new era of computing upon us. Software
globalization will require reliable software systems.
CSE
is not an “all or nothing” technology. It should be used to complement other
software development life cycles (Oshana, 1998). It is recommended that CSE be
phased-in with the current software development methodology. This will make
adoption much easier (Deck, 1994). For example, Microsoft could use a
“phased-in version of CSE” for their current Windows 2000 operating system.
First
Microsoft could replace beta testers with team reviews of program code (i.e.
verification of correctness). Microsoft, along with other software companies,
tends to market error-ridden programs and then uses consumers as their free
developer testers. What makes this comical is that as consumers we purchase
those software problems. In addition, we spend endless hours explaining the
problem or trying to resolve it.
Another
phased-in approach Microsoft could use is to perform the statistical testing on
the marketable version of the Windows 2000 operating software. Introducing
reliability measurements is an easy extension after the commitment to
statistical testing has been made (Henderson, 1995). This way Microsoft could
proactively find errors instead of relying on consumer usage. After finding
these errors and correcting them, Microsoft could continue the norm of
providing updates or software patches.
Now,
Microsoft is not the only supplier of COTS software that could benefit from
CSE. There are numerous companies. However, Microsoft provides the operating
system for which other COTS software is developed. They are the foundation and
this foundation should have much better quality than what is currently
released. Developing high quality software should be in every company’s best
interest. CSE has plenty of benefits that outweigh any obstacle or excuse.
Conclusion
If anything is new to CSE,
it is the attitude that near zero-defect software is possible (Deck, 1994).
This paper provided an in-depth review of CSE including a description of what
it is, the advantages and disadvantages of using it, and real world examples of
its use. In addition, this paper provided some thoughtful points on how CSE
could be “phased-in” to develop better COTS software.
CSE is still thought of as
some mythical software development process. However, more companies need to
make quality software besides innovation as top priority. Further research is
still needed on CSE. More companies need to pilot and publish how CSE was
integrated in their current software development process. Once this happens,
CSE will not be this unachievable mythical creature.
It
is sad fact that 60 percent of the U.S. workforce is fixing software errors
(Chapman, 2000). Developing high quality software that is within budget and
does not require a massive customer service staff should be on any software
company’s objective list. CSE is no silver bullet, however it does provide a
more scientific and measurable process that takes the art out of software
development.
References
Barron,
C. A. (2000, December 6). High Tech's Missionaries of Sloppiness. Salon.
Retrieved March 12, 2001, from the World Wide Web: http://www.salon.com/tech/feature/2000/12/06/bad_computers/index.html.
Becker,
S. A., Deck, M., & Janzon, T. (1996, October 29-30). Cleanroom and Organizational Change. Paper presented at the Proc.
14th Pacific Northwest Software Quality Conference, Portland, Oregon.
Chapman,
G. (2000, November 27). No 'Silver Bullet' for Software's Growing Complexity. LA Times. Retrieved March 12, 2001, from the World Wide Web: http://www.latimes.com/business/columns/dnation/20001127/t000113753.html.
Deck,
M. (1994, October). Cleanroom Software
Engineering: Quality Improvement and Cost Reduction. Paper presented at the
Proc. 12th Pacific Northwest Software Quality Conference.
Deck,
M. (1996, October 10-11). Data
Abstraction in the Box Structures Approach. Paper presented at the Proc.
3rd International Conference on Cleanroom Software Engineering Practices,
College Park, Maryland.
Deck,
M. (1996, October 29-30). Cleanroom
Review Techniques for Application Development. Paper presented at the
International Conference on Software Quality, Ottawa, Ontario, Canada.
Deck,
M. (1997, May 27-30). Cleanroom Software
Engineering Myths and Realities. Paper presented at the Quality Week, San
Francisco, CA.
Deck,
M., & Whittaker, J. A. (1997, May 5-7). Lessons
Learned from Fifteen Years of Cleanroom Testing. Paper presented at the
Software Testing, Analysis, and Review (STAR), San Jose, CA.
Foreman,
J. (1997, October 27). Cleanroom Software Engineering: Software Technology
Review. Carnegie Mellon Software
Engineering Institute. Retrieved
March 24, 2001, from the World Wide Web: http://www.sei.cmu.edu/str/descriptions/cleanroom_body.html.
Glass,
R. L. (1999, February). The Realities of Software Technology Payoffs. Communications of the ACM, 42(2), 74-79.
Grillo,
P., & Tewari, R. (1997). Implications of Using Cleanroom Software
Engineering to Develop Highly Reliable Three-Tier Client/Server Applications. Baylor University's Hankamer School of
Business. Retrieved March 10, 2001,
from the World Wide Web: http://hsb.baylor.edu/ramsower/ais.ac.97/papers/grillo.htm.
Head,
G. E. (1994, June 1). Six-sigma Software Using Cleanroom Software Engineering
Techniques. Electric Library:
Hewlett-Packard Journal. Retrieved
March 11, 2001, from the World Wide Web: http://wwws.elibrary.com/.
Henderson,
J. (1995, May). Why Isn't Cleanroom the Universal Software Development
Methodology? U.S. Air Force's Software
Technology Support Center.
Retrieved March 11, 2001, from the World Wide Web: http://stsc.hill.af.mil/crosstalk/1995/may/cleanroo.asp.
Kelly,
D. P., & Oshana, R. S. (1996, November). Integrating Cleanroom Software
Methods Into an SEI Level 4-5 Program. U.S.
Air Force's Software Technology Support Center. Retrieved March 11, 2001, from the World Wide Web: http://stsc.hill.af.mil/CrossTalk/1996/nov/xt96d11f.asp.
Morris,
J. H. (2000, April). A High Dependability Computing Consortium. Carnegie Mellon School of Computer Science. Retrieved March 11, 2001, from the World
Wide Web: http://www.cs.cmu.edu/~jhm/hdcc.htm.
Oshana,
R. S. (1998, November-December). Tailoring Cleanroom for Industrial Use. IEEE Software. Retrieved March 26, 2001, from the World Wide Web: http://computer.org/software/so1998/s6046abs.htm.
Oshana,
R. S., & Linger, R. C. (1999). Capability Maturity Model Software
Development Using Cleanroom Software Engineering Principles - Results of an
Industry Project. IEEE Online: 32nd
Hawaii International Conference on System Sciences. Retrieved March 26, 2001, from the World
Wide Web: http://computer.org/proceedings/hicss/0001//00017/00017042abs.htm.
Stavely,
A. M. (1999, March 22-24). High-Quality
Software Through Semiformal Specification and Verification. Paper presented
at the 12th Conference on Software Engineering Education and Training, New
Orleans, Louisiana.
Appendix A
CSE Model
Figure 1. CSE Model (Oshana & Linger, 1999)