Agile software development: it's about feedback and change

GUEST EDITORS’ INTRODUCTION

Agile Software

Development:

It’s about

Feedback

and Change

ilarities, these practitioners authored their method-

ologies independently.

Underlying values

In February 2001, 17 of these practitioners and

authors met in Snowbird, Utah, to discuss the fun-

damental similarities of their experiences and their

then-called lightweight methodologies.

Finding that

their work habits had much in common, they rec-

ognized that the lightness in their work process pro-

vided a means to achieve the more relevant end:

customer satisfaction and high quality. They cate-

gorized their methodologies as “agile”—a term with

a decade of use in ﬂexible manufacturing practices.

The participants wrote the “Manifesto for Agile

Software Development” (www.agilemanifesto.org),

which describes the four comparative values under-

lying the agile position:

• individuals and interactions over processes and

tools,

• working software over comprehensive docu-

mentation,

• customer collaboration over contract negotia-

tion, and

• responding to change over following a plan.

These four comparative values recognize the impor-

tance of the items on the right of each comparison,

Currently, the focus is on determining how to

blend agile methodologies with plan-driven

approaches to software development.

Laurie

Williams

North Carolina State

University

Alistair

Cockburn

Humans and

Technology

gile software development has hit a nerve

in the software development community.

Some people argue vociferously for it, oth-

ers argue equally against it, and others are

working to mix agile and plan-driven

approaches. Many more people wonder just what

agility is.

WHAT IS AGILITY?

In many development methods, particularly plan-

driven ones, work begins with soliciting and docu-

menting a complete set of requirements. Beginning

in the mid-1990s, many found this initial require-

ments documentation step frustrating and, perhaps,

impossible.

Both technology and the business envi-

ronment kept shifting during the project, and both

the requirements and project plans got out of date

within even relatively short projects. Customers

became increasingly unable to deﬁnitively state their

needs up front.

As a result, practitioners developed methodolo-

gies and practices to embrace, rather than reject,

higher rates of change. These methodologies were

developed on three different continents: the

Dynamic Systems Development Method in Europe;

Feature-Driven Development in Australia; and

Extreme Programming, Crystal, Adaptive Software

Development, and Scrum in the US. Although their

practices and philosophies shared fundamental sim-

Published by the IEEE Computer Society

40 Computer

but assert the dominance of the ones on

the left.

Empirical versus defined processes

Another point of commonality for all

agile methods is the recognition of software

development as an empirical (or nonlinear)

process. In engineering, processes are classi-

ﬁed as deﬁned or empirical.

A deﬁned process

is one that can be started and allowed to run

to completion, producing the same results

every time.

Assembling an automobile is such

a process. Engineers can design a process to

assemble the car and specify an assembly

order and actions on the part of the assembly-line

workers, machines, and robots. Generally speak-

ing, if the manufacturing process follows these pre-

deﬁned steps, it will produce a high-quality car.

Software development cannot be considered a

deﬁned process because too much change occurs

during the time that the team is developing the

product. It is highly unlikely that any set of prede-

ﬁned steps will lead to a desirable, predictable out-

come because requirements change, technology

changes, people are added and taken off the team,

and so on.

In an engineering context, empirical processes

necessitate short “inspect-and-adapt” cycles and

frequent, short feedback loops.

These short

inspect-and-adapt cycles can help agile method-

ologies better handle the software industry’s con-

ﬂicting and unpredictable demands.

Other empirical methods that are in synch with

the Agile Manifesto can also be considered agile.

Agility, would-be agility, and various competitive

counterparts have become hot topics in conferences

and magazines—including IEEE Computer Society

publications.

5-8

There are now both books and

annual international conferences on the topic.

CURRENT CONVERSATIONS

As practitioners have gained experience with the

use of agile methods, the focus of their conversa-

tions about this approach to software development

has changed.

Suitable circumstances

In 2001, the conversation around agile was: What

is it really? Is it old, new, a fad, crazy, or what?

We found that agile development is not particu-

larly new. Software developers have been sporadi-

cally using the techniques since at least the 1960s.

What is new, if anything, is the bundling of the tech-

niques into a theoretical and practical framework

and the strong, sometimes vehement, declaration

of their importance.

In 2001, enough sample projects had been col-

lected to make the case that agile development

works under “suitable circumstances.” These cir-

cumstances generally can be categorized as non-

safety-critical projects with volatile requirements,

built by relatively small and skilled colocated teams.

Scalability

In 2002, the agile conversation focused on two

questions: How do we make it scale beyond these

“suitable circumstances?” Does agility contradict

the Software Engineering Institute’s Capability

Maturity Model and ISO 9000 rules?

A few projects of several hundred people have

used the agile framework succesfully.

However, dur-

ing 2002, agile practitioners and other researchers

concluded that the agile value set and practices best

suit colocated teams of about 50 people or fewer

who have easy access to user and business experts

and are developing projects that are not life-critical.

Although the agile value set might be adopted (and

may be adapted) under other circumstances, the

boundary conditions for truly agile behavior seem

to be fairly set.

A close but liberal reading of the CMM reveals

little that would prevent CMM-concerned teams

from using agile methods.

However, some com-

mon CMM practices are at odds with the agile

value set because of their reliance on processes,

plans, and delivering to original contract.

Generally, people willing to spend the money on

CMM certiﬁcation are less interested in the agile

value proposition, while those needing agility for

business reasons are less interested in getting CMM

or ISO 9000 certiﬁcation.

Adaptability

In 2003, the conversation has changed again:

Can we blend selected agile practices with our “reg-

ular” practices? How much change is necessary

when transitioning to and using agile methods?

How can agile practices improve the quality of our

products?

Development groups are trying pair program-

ming without any other agile practices, test-first

development without pair programming, and short

delivery cycles without pair programming or test-

ﬁrst development, and trying to determine how to

get any of these practices to work with interna-

tionally distributed teams. Researchers worldwide

are gathering these agile subset experiences and

designing and staging additional experiments to

A point of

commonality for

all agile methods

is the recognition

of software

development

as an empirical

process.

empirically assess the efﬁcacy of isolated practices

or agile subsets.

We see agile philosophies and practices dispersed

over a wider range of people, projects, and organi-

zations. For example, many CMM and ISO 9000

organizations now think that partial adoption of

agile practices, when handled with care, might

increase their efﬁciencies without damaging their

certiﬁcations.

Making changes

As agile development spreads from the early

adopters to a larger population of developers and

development managers, the question of change is

coming to the fore.

Agile development carries with it the implication

of changing work habits. For those who have not

used agile techniques, the change involves trying

out and selecting appropriate ones. For those

already using some of the techniques, the change

involves adjusting and improving their use over

time. Some proportion of the development com-

munity has no interest in change, however, which

may impose an inevitable limit to the penetration

of agile techniques.

Agile development can affect the power structure

within an organization because it spreads out the

decision-making authority. Executives are expected

to make and take accountability for business-level

decisions, deferring to the developers on technical

issues such as development techniques and time

estimates. The developers are then accountable for

these issues.

This decision-making approach differs from

what we see in many organizations. In some, the

programmers make many key decisions, including

those pertaining to business, process, and systems

requirements, while their managers either happily

or reluctantly go along for the ride. In other orga-

nizations, programmers have limited decision-mak-

ing authority, and they are treated as little more

than clerical workers.

In either case, the move to agile development,

with its reallocation of decision-making authority

and accountability, can cause consternation. We

encounter managers who report that they consider

agile development to be a license for developers to

hack, but we also encounter developers who con-

tend that agility is merely a ploy to let managers

micromanage their work. Just how to handle this

power shift remains an open question.

Finally, some question the ability of agile prac-

tices to improve product quality. Agile practices

stress early involvement of test groups and the need

for rapid feedback, but they don’t say much

about the relationship between the develop-

ment and quality assurance groups. Integrat-

ing quality assurance into projects to enhance

their agility requires making changes in the

relationship between these groups and even

in hiring practices.

IN THIS ISSUE

The articles in this issue capture some

results from earlier conversations and set the

stage for new ones.

In their short and colorful “Agility through

Discipline: A Debate,” Kent Beck and Barry Boehm

suggest that agile and plan-driven development

exist in different ecological niches and that each

will thrive or dominate depending on the “current

weather conditions.” Somewhere between the ele-

phant, the monkey, the mouse, and the dinosaur,

their debate conveys the message that both

approaches require discipline to work—although

with some different shadings of what discipline

means—and that each works well in specific

circumstances.

Craig Larman and Vic Basili’s article, “Iterative

and Incremental Development: A Brief History,”

traces this cornerstone agile practice back to the

1950s and even before. They cite NASA’s 1957

Mercury project, which used half-day increments,

and a 1969 report to IBM management that recog-

nized “the futility of separating design, evaluation,

and documentation processes in software-system

design.” Their article includes a quote from Gerald

Weinberg, who writes of an experience at about this

time “where the technique used was, as far as I can

tell, indistinguishable from XP.”

Barry Boehm and Rich Turner examine the agile

versus plan-driven debate and provide recommen-

dations for how and when to mix the two

approaches. In “Using Risk to Balance Agile and

Plan-Driven Methods,” they highlight both the

home grounds and risks associated with each

approach. They describe how they would choose

which approach to use and under what circum-

stances they would mix them, highlighting their

ideas with three sample projects that should use

different combinations of agile and plan-driven

practices. They characterize the projects using a

ﬁve-point radial diagram that isolates important

project features driving this decision. The diagram

also captures some part of the organization’s abil-

ity to undertake change.

Boehm and Turner list three categories of risk

that can affect the choice of how heavily to bias a

June 2003 41

Integrating

quality assurance

into projects

to enhance

their agility

requires making

changes.

42 Computer

project toward the agile or plan-driven side:

• risks stemming from an agile approach:

scalability, criticality, design simplicity,

staff churn, and staff skills;

• risks stemming from a plan-driven

approach: emerging requirements, con-

stant change, need for rapid results, and

staff skills; and

• general environmental risks: technology

uncertainties, diverse stakeholders, and

complex systems.

In listing these risks, they establish a frame-

work for examining the project experiences and

characteristics of various development approaches,

a framework echoed in the other three articles.

In “Developing Complex Projects Using XP with

Extensions,” Martin Lippert and his coauthors

describe how they have dealt with circumstances

encountered on projects using (mostly) Extreme

Programming. There is an echo of the Boehm-

Turner list in this work, even though the authors

arrived at their cyclical approach to software devel-

opment independently, based on specific project

experiences.

The authors discuss how they handled diverse and

conﬂicting interest groups driving a project, lack of

testing on the customer’s side of the equation, com-

plex systems and imprecise requirements, long-term

planning, dependencies among nonprogramming

tasks, and the need for requirements deﬁnition at

least for funding purposes. It is refreshing that they

identify the special skills the people in key roles

need, including the product manager, the author-

critic, and even the customer. Their experiences pro-

vide additional practices for agile developers to

consider using and demonstrate the beneﬁts of mix-

ing selected traditional approaches with standard

agile and situationally derived practices.

In “Introducing an Agile Process to an Organi-

zation,” Mike Cohn and Doris Ford describe their

experiences with introducing Scrum into seven orga-

nizations over a four-year period and reﬂect on what

they learned along the way. They describe watch-

ing some developers add formal documents back

into their work practices—“we invariably found

programmers who enjoy producing noncode arti-

facts far more than they are willing to admit”—and

recall other overzealous developers who view agile

development as meaning they must not think ahead.

In light of the Beck-Boehm debate around disci-

pline, Cohn and Ford provide an interesting descrip-

tion: “They did not have the discipline XP requires

and, while paying lip service to XP, they were actu-

ally doing nothing more than hacking.” These

authors discuss their experiences with some Boehm-

Turner issues such as distributed development and

describe other issues that popped up on their pro-

jects, such as encountering developer resistance to

lightening the process, working with the human

resources department, using testers, and managing

the project and its interfaces to other groups.

Finally, in “Migrating Agile Methods to

Standardized Development Practice,” Mark Lycett

and his coauthors address a new wrinkle to intro-

ducing agile processes into an organization: work-

ing in an ISO 9000 or CMMI environment or one

in which a large corporation has mandated that all

projects must use a common process. They make

clear the need for a company to establish quality of

product, process, and service while still letting dif-

ferent projects work in the various ways that suit

them best.

Reporting on their experiences with a large com-

pany, these authors describe their particular

approach to project-speciﬁc process tailoring: a set

of process “patterns” that can be mapped to the

CMMI framework. For ISO 9000, they adopted a

pattern-based framework “to supplant repeatabil-

ity with consistency, while providing the audit trail

necessary for assessment.” For good project-

speciﬁc results, they revisited their use of the pat-

terns after each iteration and found that “making

the decision process visible is extremely valuable

because it forces management to actively determine

whether an activity or artifact is both necessary and

sufﬁcient.” Echoing and extending the ﬁrst Agile

Manifesto value, they describe their framework as

“suggesting suitable techniques for collaboration,

interaction, and communication.”

Lycett and his coauthors offer their own resolu-

tion of the agile versus engineering debate: “Agile

process does not ﬂy in the face of engineering prac-

tice. If used thoughtfully, it provides a clear man-

date for making engineering practice lean and well

focused.”

t is a pleasure to introduce these articles in this

special issue. They capture the state of the cur-

rent conversation, and we hope they serve as the

catalyst for future ones. ■

References

1. J. Highsmith, Agile Software Development Ecosys-

tems, Addison-Wesley, 2002.

Working in an

environment in

which all projects

must use a

common process

adds a new wrinkle

to introducing

agile processes.

2. M. Fowler and J. Highsmith, “The Agile Manifesto,”

Software Development, Aug. 2001, pp. 28-32.

3. B.A. Ogunnaike and W.H. Ray, Process Dynamics,

Modeling, and Control, Oxford Univ. Press, 1994.

4. K. Schwaber and M. Beedle, Agile Software Devel-

opment with SCRUM, Prentice Hall, 2002.

5. B. Boehm, “Get Ready for Agile Methods, with

Care,” Computer, Jan. 2002, pp. 64-69.

6. A. Cockburn and J. Highsmith, “Agile Software

Development: The People Factor,” Computer, Nov.

2001, pp. 131-133.

7. J. Highsmith and A. Cockburn, “Agile Software

Development: The Business of Innovation,” Com-

puter, Sept. 2001, pp. 120-122.

8. L. Williams, “The XP Programmer: The Few-Minutes

Programmer” (guest editor’s introduction), IEEE

Software, May/June 2003, pp. 16-20.

9. R. Crocker, Large-Scale Agile Software Development,

Addison-Wesley, 2003.

10. M. Paulk, “Extreme Programming from a CMM Per-

spective,” IEEE Software, Nov./Dec., 2001, pp. 19-26.

Laurie Williams is an assistant professor of com-

puter science at North Carolina State University.

Her research interests include agile software devel-

opment methodologies and practices, software reli-

ability, software testing, and software engineering

for secure application development. She received a

PhD in computer science from the University of

Utah. Williams is the coauthor of Pair Program-

ming Illuminated and Extreme Programming Per-

spectives (both Addison-Wesley, 2003). She is a

member of the IEEE and the ACM. Contact her at

[email protected].

Alistair Cockburn, a coauthor of the “Manifesto

for Agile Software Development,” received a PhD

from the University of Oslo on the subject of peo-

ple and methodologies in software development.

He has published widely on formal speciﬁcation,

object-oriented design, and development processes,

but is most widely recognized as the author of Writ-

ing Effective Use Cases and Agile Software Devel-

opment. Many of his publications are available

online at alistair.cockburn.us. Contact him at

[email protected].

EEE Pervasive Computing

delivers the latest peer-reviewed

developments in pervasive, mobile, and ubiquitous computing

and acts as a catalyst for realizing the vision of pervasive (or

ubiquitous) computing, described by Mark Weiser nearly a

decade ago. In 2003, look for articles on

•

Security & Privacy

•

The Human Experience

•

Building Systems That Deal with

Uncertainty

•

Interfacing with the Physical World

SUBSCRIBE NOW!

http://computer.org/pervasive

M. Satyanarayanan

Carnegie Mellon Univ. and Intel Research Pittsburgh

Associate EICs

IEEE Pervasive Computing

Editor in Chief

Roy Want, Intel Research; Tim Kindberg, HP Labs;

Deborah Estrin, UCLA; Gregory Abowd, GeorgiaTech.;

Nigel Davies, Lancaster University and Arizona University