Chapter 2: Software Project Characteristics - Software Development Failures [Electronic resources] : Anatomy of Abandoned Projects نسخه متنی

Chapter 2: Software Project Characteristics

Software is so crucial to the operations and management of organizations worldwide that it is inconceivable that any organization could compete successfully in the information economy without substantial investments in software projects. Investments in software, however, come with significant risks of failure. Indeed, it is common knowledge in the information technology (IT) industry that software projects are over budget and behind schedule and often have less functionality in far more cases than IT professionals and management find acceptable. The questions that need to be answered include the following: What factors intrinsic to software development projects make them particularly susceptible to such fiascoes? How do such factors combine to make software projects vulnerable to cancellation and other problems, such as cost overruns and schedule delays? This chapter looks closely at some of the characteristics of software development that can affect software project outcomes. In addition, it reviews a number of studies examining the assessment and management of the risks and uncertainties in software development. It compares the results of the risk-assessment studies with factors that have been identified as contributing to the failure of software projects and points out underlying similarities between the two sets of results.

Software production is a complex and exceedingly difficult creative process. It involves the sustained collaboration of disparate groups of stakeholders to achieve performance objectives intended to satisfy organizational requirements. The intrinsic properties of software projects include, for example, project size and complexity, problem formulation and abstraction, project-team composition, and project risks and uncertainties. This list is not meant to be exhaustive, and some items overlap. For instance, unwieldy project size and complexity contributes substantially to the risks associated with the development effort. Also, technical and technological issues involved in software development are not specifically mentioned but can be subsumed under project size and complexity and project-team composition. These characteristics of software projects play a significant role in the software crises that form the subject of this book. Let us examine each of these characteristics to determine the nature of their contributions to the software development problem.

Project Size and Complexity

The difficulty of software design is heightened as the size of the project increases, and with increasing project size comes a level of complexity that is orders-of-magnitude higher. Software design is concerned with the creation of artifacts or products that are robust and reliable and meet the functional specifications of the user in the problem-solving context (Winograd 1996). Computer programs are the implementation or realization of the design objects. The processes of creating the design and of implementing the design in a programming-language context add to the complexity of the endeavor. The complexity stems partly from the difficulties in comprehending the various facets of the design problem in order to come up with a robust and reliable design. Complexity also stems from the difficulty in coordinating the activities of various professionals engaged in the enterprise. The orders-of-magnitude increase in the level of complexity of a software project often overwhelms our capacity to fully comprehend all the possible states of the designed system. This failure to comprehend all the states of the design invariably has consequences for the implementation of the system, as is often revealed by testing of the system. Thus, as Fred Brooks ([1975] 1995) has said, the complexity of software is an essential property of the entity that cannot be stripped away or minimized through abstraction.

The size of the project invariably increases the complexity dimension of the problem. Large software projects are orders-of-magnitude more complex than small projects, in part because they often require more personnel. The coordination and communication required for the collaboration of the various groups add another level of complexity to an already-complex creative process. It is common knowledge in the computer industry that small software projects are often quite successful in meeting schedule and cost estimates but that the "risks and hazards" of cancellation and/or major delays and cost overruns tend to rise significantly as the size of the project goes up (Capers Jones 1995). This is not to suggest that project complexity is a function of or that it correlates with project size. Small projects can also be rather complex, depending on the types of problems they address. But in general, large projects are especially complex and difficult.

A number of design techniques in use are intended to reduce the complexity of software projects. Notable among these approaches are concepts such as top-down hierarchic design. This structured technique uses top-down design in an attempt to reduce the intrinsic complexity of systems problems by imposing a hierarchic structure. The architectural schemes available to the software designer include the ability to derive these hierarchic structures from complexity. The interactions that take place in hierarchic structures can be distinguished to reflect intersubsystems interactions as separate from intrasubsystems interactions (Simon 1984). This ability to create subsystems out of complex systems makes it possible to focus attention on each individual subsystem in order to understand it. But we are still faced with the need to understand the interactions among subsystems in order to fully comprehend the entire complex system. Thus the hierarchic systems demand a great deal of learning and comprehension on the part of the software designer in terms of the designer's ability to visualize the whole picture posed by the design problem. David Parnas (1985, 16) has written ably about this problem, stating that "even in highly structured systems, surprises and unreliability occur because the human mind is not able to fully comprehend the many conditions that can arise because of the interactions of these components."

Finally, the complexity that is a permanent feature of software projects is unlike the complexity in natural systems, because it is not redundant; no aspect of its structure can be inferred from any other. Indeed it is what Brooks (1987, 1996) insightfully labels arbitrary complexity. It is the kind of complexity that is most difficult to deal with because its various interactive states and levels are hidden and must be painstakingly uncovered one layer and type of interaction at a time. This is the ultimate challenge software developers face, and as projects tackle more and more intricate and demanding problems, we can expect the size and level of complexity of the projects as well as the difficulties associated with them to increase. Parnas (1985, 18) explains why this is the case: "All of our experience indicates that the difficulties in building software increase with the size of the system, with the number of independently modifiable subsystems, and with the number of interfaces that must be defined. Problems worsen when interfaces may change. The consequent modifications increase the complexity of the software and the difficulty of making a change correctly."

Experience thus far seems to show a failure of the modernist approach to software development to "deliver the promises of high rationality" (Robinson et al. 1998, 367). Indeed Brooks (1987, 1996), Parnas (1985), and others—including Robinson et al. (1998), Barwise (1989), Fetzer (1988, 1989), and Leith (1986)—have forcefully cautioned against such utopian ideals of rationality in software development, because of our limited ability to fathom, much less describe or model, all the subtleties of a problem in a software product. Hence the persistent need to revisit even the "successfully" implemented and operational software to "correct" the errors or problems and omissions in the "finished" product. The formal methods and tools may prove inadequate to the complexities of the task. At other times the software problems may result from the inability of developers to fully and accurately grasp the essence of the problems from the beginning; this inability lays the foundation for the problems that will later manifest themselves—even in software projects that do make it to completion. Furthermore, the inappropriate or inaccurate use of the most suitable formal methods and tools applied to the right software task may generate its own set of development problems, even in the absence of any difficulties in describing the requirements for the software. The human and organizational aspects of software development are perhaps the most compelling yet stubborn dimension of the entire process and provide the most convincing explanations as to why software development as a process cannot be completely rationalized to overcome the problems outlined above.