| DHQ: What are the
"five core metrics" featured in your new book? When in the development process
do you take those measurements? Putnam: The
five core metrics are size, schedule time, effort (cost), defects, and process
productivity. You use them from the start of and then throughout the life of the
development process, whenever necessary and appropriate. Myers:
In the first two phases of the Unified Process, Inception and Elaboration, one
of the purposes is to figure out sufficiently what is to be done, in order to
attach a size estimate to the project. So, thinking about what is to be done (especially
in terms of size) goes all the way back to the beginning.
DHQ: Who on the development team is responsible for
recording metrics? Does it require additional staff? Putnam:
The project manager is responsible overall. He or she usually has a quality assurance
team as part of the development team. This is often a very good place to put the
metrics collection and analysis function. It can be done as a part-time effort.
It usually does not require a full-time dedicated person. Tools are useful for
recording and processing the metrics data. The tools can be homegrown using spreadsheets.
Also, there are excellent commercial tools that have been specially developed
for this purpose. Such tools aid in collecting the metrics data and also provide
analytical capabilities to interpret the data and tell the story the metrics are
designed to convey.
DHQ: How often do you record
the metrics? Putnam: Monthly or weekly during
development, so you have a good time history of what went on during the project.
DHQ: How often and when do you do analysis work with the
metrics? Putnam: Do analysis at the start
of the project, using historic data and estimates of the key parameters for the
new project. This is related to the estimate for the new project. Update this
estimate whenever the scope or other key factors of the project change. Use the
monthly or weekly metrics data to do comparison analysis as soon as the project
is underway. By the time the project is roughly 25 percent complete in schedule,
you can compare actuals against the plan and make an adaptive update forecast
if actuals are deviating significantly from plan. At the end of the project, capture
all the key information relating to the size, schedule, effort, and defects. Use
this to determine the process productivity achieved. Store this in your metrics
repository to use on new projects coming along. Ware: One
addition that I would make to what Larry said is something about the psychology
of data collection. Management should make it clear that this data is being collected
for metric purposes, not for personnel use. People are willing to collect data
honestly for estimating and control purposes, but if they suspect it is to have
something to do with personnel appraising, they begin to fudge the data and consequently
to distort the data for metric purposes.
DHQ: What's
new about relating metrics to the Unified Process, and why isn't it enough to
follow the process without metrics? Putnam:
Metrics can be a part of any development methodology and should be. The originators
of the Unified Process did not make metrics a part of their original creation.
However, the company they are affiliated with, Rational, has been integrating
metrics with a number of its products and is beginning to infuse its thinking
with metrics concepts. Metrics ideas are important to the management of development
because they allow resource requirements for development to be integrated into
the planning and control process that is at the heart of management. This permits
you to put the right number of staff on the project at the right time; it permits
the prediction of when defects will occur and how many, so defect elimination
is timely; it allows you to project ahead to determine when the defect discovery
rate will be low enough that the product can be released for use by the end user.
These metrics functions are important to the planning and control of any development
process. There is a significant void in development when metrics are not used.
DHQ: Won't keeping track of metrics make it more difficult to follow
a formal software development process? Putnam:
Keeping track of metrics during development is not an onerous task. It can be
handled part-time by project support people (usually in quality assurance). The
volume of data collected is very modest when focused on the five core metrics
we discuss in our book. There are good commercial tools available that collect
data, help with analysis, and plan and control the development process, so the
time and effort spent servicing the metrics work is very modest and not costly.
Myers: It is perfectly true that collecting metric data takes a
little time and making use of it takes a little more time. But it is customary
on a planet of limited resources—limited daytime, limited lifespans, and
so on—to try to get work done within some limits. In fact, we have a name
for it: competition. And those who don't come close to competition-set limits
are removed from business (and even government). In software development, the
name for this set of activities is metrics. To the extent that software development
is an economic activity, not a weekend pastime, it has to base itself on metrics,
and most organizations do use some kind of metrics. The issue, really, is whether
the metrics are good or "seat of the pants." Part of making them good is the realization
that they are tied into process, models, and tools.
DHQ:
Don't metrics bog-down the speed and creativity of software development? Putnam:
No. Actually, they speed it up because there are fewer false starts and
fewer rework cycles. They can be handled as a by-product of the development process.
Moreover, they provide valuable insights into what is happening on the project
while it is under way, and they afford an opportunity to make timely changes to
the project by providing near-real-time observation of problems that may ultimately
cause cost and schedule overruns. Myers: I think you could
even put the issue in the reverse form. Lack of metrics, and of the process, models,
tools, and so on that are concomitants of metrics, causes projects to fail or
to overrun arbitrarily set cost and schedule limits.
DHQ:
If my group hasn't kept track of metrics in the past, what can I do as an individual—and
what must my group do—to start using metrics? Putnam:
Any innovation has to start sometime. Look for the opportunity and push to get
it started on your project. Point out the benefits of early visibility of problems
and of having better plans for avoiding the problems and bottlenecks that happen
without the use of metrics. Work on the culture change that may be necessary to
convince team members that metrics will help them avoid death marches, which are
common in the absence of good metrics use. Myers: In addition
to Larry's observation that an individual can encourage the use of metrics, you
need to convey to management on a broader scale, as we do in the book, the advantages
that metrics, when integrated into process, models, and tools, can bring to the
organization. For an indication of the meager interest that management has displayed
in the subject, note that the Harvard Business Review seldom deals with the management
of software development. Yet it is not only costly in its own right, but failure
to develop effective software impinges on every other area of business.
DHQ: Compare this book with your previous collaborations—what's new?
Putnam: This book is less about the mechanics of estimating and
measurement and much more about how metrics fit into the development process and
yield real benefits to the development team and the paying customer. It is a more
integrated view of metrics as an essential part of the development process. Myers:
I would add to Larry's observations that the present book spends more time on
the first two phases, Inception and Elaboration, and the fourth phase, Transition,
than our earlier books did. Our first book, Measures for Excellence, started
at the point where some experienced people sat down to estimate the size of a
system that had already been outlined. How the system got to the point of being
sized, we did not deal with very much. Similarly, we largely stopped at the end
of the Construction phase, saying little about the problems of getting a system
into operation in the field.
Secondly, the earlier
books largely assumed that there would be a software process of some sort and
that a given organization would be able to repeat it. Without repetition capability,
of course, an estimate is futile. Organizations tend to go on doing what they
are already doing, of course. In this book, we have been much more explicit about
process. It has now been more firmly established that there is process, and that
the process can be vastly improved by consciously taken steps. We add that Putnam's
methods can measure this improvement, which supports the motivation to improve
the process.
Finally, in Part IV of the book,
we go beyond the level of the individual project and consider department- or company-level
problems that are amenable to effective software management.
DHQ: What projects are you working on now, together or individually?
Putnam: I'm working on updates to our Web page, articles for the
Web page, new copy for the latest version of our SLIM tools (V 6.0) and a pitch
at the 2nd International Conference on Process Improvement in mid-November. In
addition, I'm trying to focus on business matters, to get back on an upswing after
three years of depression in the economy. Things look better now, and we need
to exploit that.
I have been interested for
a long time in whether or not our estimating approach and key algorithm (software
equation) would work in other domains. I think we are quite confident that they
will. I want to pursue this idea, get some data, test it more thoroughly than
we have already, and perhaps turn it into a usable tool. Myers:
However, I may be in the process of arriving at an idea—something to the
effect that software development is a good deal bigger than just writing code.
In Five Core Metrics, we developed the relationship between metrics and
process, plus models, plus tools. In another book, we might extend all that to
the field of negotiation. At the front end, prior to the inception phase, we might
extend software development back into the reengineering area, that is, we might
overhaul existing practices first.
Looked at
in this way, it becomes evident that this whole has to exist in an economic framework
measured and controlled by some kind of metrics. This whole is an obvious concern
of top management. And while some in that field are meeting the challenge, many
are not.
Well, I don't know if I know enough
to cover this broad scope, but it won't do any harm to let the bigger idea germinate
for a while. In effect, I am in the early stages of thinking about the next book,
but I am not yet ready to spill the beans, because most of the beans are not in
the pot yet.
DHQ: Thanks, Larry and Ware! | 
