| DHQ: In your new book,
Best Practices for the Formal Software Testing Process, you write that it
is the book you wish you'd had at the start of your testing career. Why is that?
What errors would you have avoided, knowing what you know now? What's your main
message to today's testers? Drabick: This
book is intended to provide a process-related look at the software and system
testing life cycle.
I wrote this book because
when I first began software testing, I had previous experience in testing complex
hardware systems. Had I just followed that model, I wouldn't have realized the
criticality of beginning test planning during requirements definition.
What
errors would I have avoided? See above. Also, I initially didn't realize the importance
of test design (i.e., identifying test cases at a high level prior to developing
the cases and writing test procedures). Fortunately, I had been introduced to
Software Quality Engineering's Systematic Test and Evaluation Process at the very
start of my software testing career, as well as being introduced to IEEE Std.
829-1991, the IEEE Standard for Software Test Documentation.
Many
people don't start out with this background. When I began my software testing
career in the early 1980Ős, software QA and testing were seen as policemen and
adversaries of the development team. Now, we realize that we should be a single
team interested in providing products to our customers with a minimum number of
defects that satisfy the user's requirements. My main message is that we test
engineers and test managers need to optimize our processes and the way we work
so that we can help our organizations deliver quality products. I believe we can
only do this if we understand our testing process and are continually working
to improve it.
DHQ: In his foreword to your book,
William Perry describes how your focus changed over the years, until you began
concentrating on the testing process. What drew you to the testing process? Drabick:
The answer to this one is simple. After attending many conferences, I realized
that process is critical. As a poster on the wall of our work area states, "Action
without process is perilous." Such a situation is also chaotic. I had also been
exposed to the importance of process by coordinating the first SEI-style self-
assessment in Eastman Kodak Company's Research and Engineering Division in 1987.
After working on a couple of process definition and improvement initiatives at
Eastman Kodak, I realized that understanding a process was the first step to effective
process improvement. What solidified my interest in the testing process was the
need to create a solid Basis of Estimate for a large program Kodak and IBM were
bidding on for the IRS. That's what led to the development of this model. By the
way, we won the contract, but that's another story.
DHQ:
One especially memorable phrase from the book is that "There is no 'one true
way' to test software." What misconception is out there—and where does
that one true way mislead testers? Drabick:
Like anything, the way you test depends on the environment you're in. If you're
in a DOD, NASA, FDA, or other highly regulated environment, you need a rigorous
testing process and approach to testing, because errors can cost people's lives.
If you're on a program working a Spiral or other iterative lifecycle, you have
need for a defined process for each iteration. In the dot-com environment, you
have to develop and test quickly to hit a very limited window of market opportunity.
In an XP environment, you're coding and testing together. Each environment demands
a different approach to testing. So a testing process model has to be tailorable.
One size DOESN'T fit all; one process doesn't apply to all projects. In some environments,
an "exploratory testing" strategy may be the best approach, as compared
to a more documented, systematic approach.
DHQ:
You also assert that testing is not a phase—that testing should start during
requirements elucidation and extend through system testing. How is your vision
of the scope of testing different from common practice, and what problems does
it solve? Drabick: There are so many visions
of the scope of testing that this question is difficult to answer without writing
a whole other book. I believe in what I like to call "lifecycle testing."
I'd like to hope that with all the work that's been done by good testing consultants
and practitioners over the years, my assertion isn't unusual. However, in a number
of companies in the commercial world, their development environment is still one
where the developers "throw products over the wall" to test engineers,
who in some cases don't even know there's a new product being worked on. (I worked
at a small company like this a few years ago. Surprising the testers with a delivery
doesn't say much for management, the test engineer, or the development engineer's
ability to communicate, but that's another issue.) If test engineers have a customer's
interests and perspective in mind, they can ask significant questions during requirements
analysis. Test engineers should also be planning their tests at that time, so
that when coding and debugging is complete, the test team has its test procedures
ready to run and the test environment in place to properly evaluate the quality
of the product. But let there be no mistake; the developers are responsible for
the quality of the product. Test and QA engineers measure the quality of the product,
but you don't achieve zero-defect code through the application of testing alone.
Note that I consider "testing" and
"SQA" to be two different disciplines. This philosophy came from the
fact that IEEE has separate standards for SQA plans (IEEE-Std-728) and Test plans
(IEEE-Std-629). I happen to have a similar process model for SQA, if anyone's
interested.
DHQ: You are a strong advocate of formal
test plans. How should testers develop a test plan? What's the key to developing
an effective test plan? Drabick: I describe
how to develop a test plan in Chapter 4 of my book, and it is also described in
books by Bill Perry, Rex Black, Rick Craig, and others. There are several keys
to developing an effective test plan, some of which are described in Chapter 4
of my book (Create Test Plan). They involve creating a workable test strategy
for your project and environment, synchronizing your test plan with the other
program plans (see Chapter 3 of my book, Review Program Plans), and in
having a testable set of requirements (see the section in Chapter 4 on Reviewing
Requirements).
Test plans
don't have to pass a weight or page-count test. If you can write a test plan for
your environment that adequately defines your test strategy and how you'll test
the functionality in your system in 5 pages or less, great. Most of the time it
takes a longer document, but the key is to use what works for you. I personally
favor a test plan written according to the format of IEEE-Std-829, but again,
use what works and what your management will tolerate.
DHQ:
In the book, you discuss ways that the formal testing process can be applied in
an Extreme Programming (XP) environment, in which the development schedule is
highly compressed. Is the formal testing process compatible with XP? How might
XP suffer from omitting parts of the process? Drabick:
For the answer to this question, you'll have to read Chapter 8 of my book (Tailoring
the Model).
Some XP projects won't have
a formal testing group, in which case they won't need a formal test process. However,
they should define the test process they are using, which will normally be a form
of "buddy testing," per Rick Craig. In XP, there isn't time for a full
test planning/test design/test case/test procedure cycle with each cycle of XP
development. A single test plan or strategy should be developed at the start of
the XP program. Then, develop your automated test scripts to support each release
cycle, keeping in mind the test strategy you identified early on. If the project
changes scope and direction as it matures, you might consider updating the strategy.
For an excellent treatise on this subject, see Lisa Crispin and Tip House's book,
Testing Extreme Programming.
DHQ:
What would you say is the biggest problem in testing today, and how can it be
remedied? Drabick: Wow, that's a tough question,
and one that I'm sure a variety of people would answer in different ways. My answer
is, our biggest problem is management that doesn't understand the importance and
the significance of testing. I just had a discussion on this subject today with
a test engineering colleague. A sharp test engineering team is too often undervalued
and unappreciated by management. Little in the way of resources or training is
provided to the test team, and then it is soundly criticized when poor products
are released (often after cutting the time allowed to perform testing, not providing
an adequate test environment, and not bringing the test team on-board when requirements
were being developed). Management conveniently forgets that the test team didn't
build the failure-prone software. That's another reason I wrote my book, so that
management could get a quick appreciation for what testing is all about.
DHQ:
What can we do to help testers and developers see themselves as a part of the
same team, rather than members of different departments? How can they work more
cooperatively? Drabick: Break down "the
walls" between groups; make sure your test team is co-located with your developers.
I remember a big project where my test team was on the seventh floor of a building,
and the developers were on the second floor. There was only a single stairway
and a slow freight elevator between the floors. Additionally, the test lab was
in the basement. Communication was rotten on that project. Staff your test team
early, and get the test team involved in the project during requirements development.
Encourage developers and the test team to share lunch-time learning sessions,
where the developers talk about their development techniques, and the test team
makes presentations about their test process, their test techniques, and new things
they are learning and doing. And have management say good things and express appreciation
for the efforts of both groups, in staff meetings. At the end of the project,
hold a project retrospective using the techniques outlined in Norman Kerth's book
with all members of the cross-functional project team.
DHQ:
What do you say to those who interpret the words "formal" and "process"
as "rigid" and "slow"? Drabick:
As I state relatively early in my book, everybody has a process. It's "the
way you do things." The question is, should you document that process? The
whole point about my process model, especially as pointed out in Chapter 8, Tailoring
the Process, is that you need to set up the process that works for you, and continually
tune it based on quantitative metrics. That removes the "rigid" objection.
One significant point of the Capability Maturity Model – Software, and the
newer Capability Maturity Model – Integration is that your process must be
continually improved based on quantitative metrics. If you have a process that's
cast in concrete, it will be ignored. Again, as was mentioned above, the process
you use has to depend on your specific environment.
DHQ:
What companies and/or types of environments are implementing the formal software
testing process you describe? Where can we find more information about their experiences?
Drabick: As mentioned earlier, every organization
has a testing process, whether it realizes it or not. The question is, how formal
and well documented is it? Both ISO and CMM/CMMI push organizations in this direction.
This would indicate that many large and medium-sized companies in the more typical,
non-Web lines of business have a formal testing group. Smaller and Web-based companies
may not have a defined process, but that doesn't mean they don't need one. I've
used the process in a division at the Eastman Kodak Company, I used it in a small
company in the medical document processing industry, and I'm working to get my
current company to adopt parts of it. To find out information, go to software
QA and testing conferences, like the International Software Testing Conference,
hosted by the Quality Assurance Institute; the STAR Conferences, hosted by Software
Quality Engineering; and Quality Week, sponsored by Software Research, Inc. I'm
sure there are others. You could also consult the QAI and SQE Websites; again,
I'm sure there are a variety of other places to look on the Web.
DHQ: What prevents testers and managers from defining a formal testing
process? How can an individual reader of this book influence the implementation
of the process you describe? Drabick: What
prevents an organization from defining a formal testing process is the fact that
until now, the definition required a lot of work, and it costs money. My book
and the embedded process model should ease this effort. Most companies working
in highly regulated industries have to have such a process, or they won't be awarded
contracts. Many subcontract development shops have to have a formal testing group
(though their process may not be defined), or they won't be awarded contracts
either. Qualifying for a subcontract may require a CMM/CMMI rating of 3 or higher.
Such a rating needs a defined testing process.
Readers
of this book can take some off-work time to download the PowerPoint or PDF charts
for the model from the Dorset House Website (www.dorsethouse.com/books/bpf.html),
and then mark up those charts to match their current process. After that, if the
reader is not a manager, he or she could approach his or her test manager with
a proposal to document the testing process. After that, it's a matter of deciding
how to improve that process using an ISO or CMMI approach.
DHQ:
Why should experienced testers, test managers, team leaders, or new testers read
this book? Who else should read it? Drabick: Experienced
test engineers, test managers, and team leaders should read this book if they
are interested in defining and documenting their existing testing process as the
first step to improving it. The diagrams in this book can also be used to teach
development personnel and managers what is involved in the testing process. There's
a lot more to it than they might expect.
New
testers will find a definition of the overall testing process in this book, so
it will provide them with a broader perspective on testing.
Other team leaders and managers in the development organization will be able to
understand all that is involved in the testing process. If you're a CEO or CFO
of a small company looking to develop high quality products, and you already have
a testing group in place or are considering implementing such a group, this is
the book for you.
I'd also highly recommend
this book to computer science or software engineering students. There aren't many
testing courses out there yet, and everyone who's going to be developing software
should know how a testing process works. Who knows, you might even decide to pursue
a career as a test engineer.
DHQ: What kinds of
college courses should consider this book for adoption? Where would this book
fit into an undergraduate or graduate software engineering curriculum? Drabick:
I believe that this book could provide the basis for an Introductory level bachelor's
quarter or semester course in software and system testing. Homework and project
assignments could be developed (linked to a programming Case Study) in requirements
review, developing test plans, developing test design, developing test cases,
developing test procedures, and then executing the tests. After the tests have
been executed, students could exchange the final tested product with other students,
who would represent users, to see if they can find additional defects that had
managed to escape detection. Since developers also understand what a formal testing
group is doing, this book would provide a basis for a supplement to a sophomore-level
course in software engineering or computer science.
DHQ:
Now that your first book has been published, what's next? What kind of work are
you doing these days? Drabick: Actually,
I'm not doing testing these days; I'm the lead SQA engineer on the Canadian Census
project at Lockheed Martin's Transportation and Security Solutions Business Group.
We're preparing a system to assist the Canadian government in their 2006 census.
It's an exciting project, and I'm working with an excellent team. We're in the
middle of our formal testing cycle right now, which means I have the opportunity
to assist our test team by reviewing their test plans and test procedures prior
to formal inspections; I also witness the formal tests.
Though
not in the near future, there may be another book in the works. Some years ago
I wrote an article for Software QA Magazine titled "Ethics in Software
QA and Testing." A publisher approached me about writing a book on that subject,
but I wasn't ready to write that book then. Given the state of industry these
days, I think that might be an interesting book to write.
DHQ:
So what does a reader do after reading your book? Drabick:
I'm going to steal a page from a number of post-conference presentations Bill
Perry used to make. What you should do, is select two or three things that you
want to do immediately with what you have learned, and begin working on those
items.
What you select will depend on who you
are and where you work. For example, if you are a junior test engineer in a company
at CMMI Level 3 or above, you should ask your testing lead or manager to describe
the formal testing process to you. You might ask him or her for any documentation
available on that process. If the process isn't documented, you might mention
this book and suggest that your group begin documenting your testing process.
That will help management and the other cross-functional groups in your project
understand just what it is your test team is doing.
As
another example, if you're a testing manager in a small company where software
gets "thrown over the wall," you should quickly download the Testing
Process Model charts from the Dorset House Website (www.dorsethouse.com/books/bpf.html),
and document your current process. Then, you might use the information in Chapter
4 to begin analyzing requirements for testability. You might also begin a process
of formally reviewing/inspecting your test plans/test designs/test procedures
with the development team. That way, both your test group and your development
group will know what you are planning to test, so that any disconnects can be
identified.
There are a number of other things
you could do, depending on who you are and the environment you work in. Hopefully,
these examples will give you some pointers as to things to consider.
|
