Software Engineering and Software Development Failures
Software development as an engineering science was born at the 1968 NATO conference convened to address the problems encountered in the software crises of the 1960s (Wasserman 1980; Blum 1994; Robinson et al. 1998). The emergence of software engineering as a systems development discipline was based on the perception that the modern scientific method—with its emphasis on formalism, rationality, and objectivity—could provide the correct approach to solving the host of problems associated with software systems development. Some of the proponents of this approach, including Dijkstra (1968), Hoare (1984), and Lehman (1989), argued that the adoption of a formal logical or structured approach would provide the "silver bullet" that would aid the discipline in overcoming the crises of systems development. For example, they argued that applying the objective standards of the modern scientific method to computer programs would enable systems developers to "intersubjectively" test their programs for correctness. Over the years, considerable energy was devoted to formulating concepts and solutions involving proofs of program correctness, stepwise refinements, functional decomposition, structured analysis and design, modular structure, information hiding, and structured programming, in a concerted effort to reduce the inherent complexities associated with formulating and solving the right systems problems (Sommerville 1997; Blum 1994; Marciniak 1994). In time, several different process models were developed to improve the practice of software development. The common theme underlying the various process models is the abstraction of the sequence of activities needed to create a software product from the user's problem description. At present, as Sommerville (1997, 2260) has rightly observed, there are "no accepted standards for describing process models." The majority of software development models in use in organizations are based loosely on some generic process model for generating the documentation required to communicate with the project team, clients, and management. Robinson et al. (1998, 366), in a critique of this era in systems development, has characterized this approach as one based on the erroneous belief that "software engineering embodies a view of the world as being composed of unitary problems, each capable of rational solution via the application of technology." This analysis of the use of "formal conceptual schema" to model the complexities of the real-world problem domain is based on the premise that natural language—with its inherent ambiguities and varied ways of describing a problem—is somewhat unsuited to handling the rigor of the "deep semantic structure of a given" problem context. Although considerable progress has been made over the last several decades in tackling systems development problems, the software crises are still with us.The view of software as a purely objective artifact or product that is created devoid of context is somewhat misleading. Context—be it organizational, cultural, managerial, or technological—shapes the software development process and the final product created. The weakness of the modernist approach to software development, with its emphasis on rationality, highlights the need to pay adequate attention to all the issues that legitimately direct the development process. The significant evolution of the tools and techniques of software development has tended to emphasize the rational or formal problem-solving perspective. However, the complexities of the cofactors that shape the "solution space" of the software development process have not diminished over the years. Hence the persistence of software crises and the continual search for ways to reduce their incidence in software development practice.In Brooks's (1987) view, no "silver bullets" exist capable of coping with the inherent difficulties associated with the nature of software and the manifestations of those difficulties in the systems development process. Brooks's realism leads him to opine that as far into the future as he can see, there exists "no simple development in either technology or in management technique, that by itself promises even one order-of-magnitude improvement in productivity, in reliability, in simplicity" in software development (p. 10). In essence, the software crises will always be with us. Thus the best we can do is to find ways to reduce their impact on the software development process. Harel (1992, 10), commenting on Brooks's pragmatic but somewhat pessimistic verdict, offers a more encouraging recitation of developments in software engineering that may provide "a vanilla framework for system modeling" and that indicate good prospects for the improvement of systems development. Robinson et al. (1998), however, see a breakdown in the "modern" scientific or formal approach as employed to tackle systems development. They argue that the formal systems derived by this rational/logical approach must eventually be "connected" to the real world they purport to represent in order to be validated. "It is at these points of connection that problems may occur," they suggest, "for this is where the implications of the epistemologies of the system and the world connect. The system's success depends on whether these epistemologies collide or co-operate" (p. 368). These are precisely the problems inherent in the nature of software that Brooks describes as the "essence" and "accidents" of software development. Adding to these problems are the inherent difficulties users have of communicating their tacit knowledge of the problem domain in the form of systems requirements to the systems developers. Moreover, as Eischen (2002, 39) observes, the users' tacit problem-domain knowledge is often "undefined, uncodified and developed over time" and is "constantly evolving," especially in a dynamic organizational environment.Software development from a postmodern perspective as described by Robinson et al. (1998, 368) is "an amalgam of various analytical, design, implementation, predictive and managerial activities, embedded in dynamic social systems, replete with already developed sites of cooperation and conflict." This view of software development explicitly recognizes that a "multiplicity of cofactors" are always at work in the software development process, any combination of which may be problematic and may be at the root of software failures, whether during development or after implementation. Thus in software development the complexity of the systems problem and its real-world organizational context must all be accounted for—from the requirements-determination phase through to the implementation. This view of software development pays the requisite attention to the management, the organizational, and the nontechnical dimensions of the software development process, areas that the modernist approach tends to ignore.In the remainder of this chapter we briefly review the extent of software development project failures, then pursue the nature of the problem. We end with a detailed description and analysis of abandoned software projects. Abandonment refers to the cancellation of the projects prior to their complete implementation in the organization, and thus can occur at any juncture in the systems development process.