Roundtable Discussion:
Dealing With Impending Disaster

edited by James Bullock, Gerald M. Weinberg, and Marie Benesh

"Usually the wisdom ‘Well, we have to at least try’ is right, but this may be one of those place[s] it wasn’t. Sort of a double-irony. It occurs to me that this dilemma has analogs in project management and product development. In product development, we sometimes have to decide about ship-or-slip:

"Do we ship with this defect, or do we slip the schedule to address it?"

—Rick Brenner

What Do You Need to Decide What to Do?

Steve Jackson
Here are some questions for project managers:

• What information would you need at what time in order to decide whether or not to pursue a tried-and-true course of action?
• How would you identify and/or recognize such information?
• How would you obtain such information in time?
• How would you convince your staff that you’re not crazy, so that they will allow you the opportunity to act in an unconventional manner (and follow you)?
• How would you know that you were successful at limiting the damage?

George Olsen
A few years ago, I read a book that looked at a variety of accidents–aviation, industrial, maritime, and so forth. One common denominator was that they tended to involve multiple "improbable" events. The second common denominator was that those involved in the impending disaster kept trying to fit what they were experiencing into a "normal" interpretation of things–instead of realizing they’d moved into extraordinary circumstances. They had all the information in front of them, but they just couldn’t "see" it. Granted, they were usually under a time pressure, but the same type of blinders seemingly apply to many projects that are disasters in slow motion.

James Bullock
If you’re referring to Normal Accidents, the book also suggests that systems that have these problems have common characteristics:

• They are closely coupled, with complex interactions between parts.
• They respond rapidly to changes.

This suggests that projects should be designed to avoid these characteristics–otherwise, those projects are just asking for disaster.

To Steve Jackson’s first question, about what information I’d need to decide whether or not to pursue a tried-and-true course of action, I’d answer that I need the history of using the tried-and-true action and the characteristics of the times it was applied. I’d need to know if it worked and how well. This information would let me know two things:

• if a real solution was available, or just a theory
• how well it worked, when it did work (potentially not well enough for all situations)

I’d need to know these details soon enough for me to take the conventional action and have the result do me some good. It’s no good putting on the brakes if it’s too late to stop.

Regarding Steve’s second question, about how I’d obtain information in time, I’d try to know what the problems might be ahead of time, and I’d develop alternative responses. Maybe I’d model the dynamics of the available responses so I’d know when it was too late to try a particular alternative (such as adding people to a late project).

I’d try to collect characteristic data about the current situation all the time. Doing so, you stand a better chance of noticing that there is a problem, and you may even have the information on-hand to formulate a solution when a problem shows up.

To Whom Do You Listen?

Dan Starr
The person who designed the Titanic was on board, though not present on the bridge when the iceberg was spotted. I wonder if he would have advised hitting it straight on rather than a glancing blow–and I wonder if anyone would have listened to him.

Jerry Weinberg
There’s a related question that’s a useful measurement I often use in projects: "Whom would they have listened to?" For example, would they have listened if God appeared and said, "Strike it head-on"? It’s a useful test because I often find projects that wouldn’t even listen to God; in such cases, I need not waste my breath. At other times, the question identifies the executive or technician who has the real power in the situation.

There Are Different Kinds of Damage

Rick Brenner
In some cases, the difference in the consequences of two choices is thought of simply in terms of how much damage we do to the company by making one choice or the other. As with the Titanic dilemma, it may also be useful to consider what kind of damage we do to the company by making one choice or the other.

Jerry Weinberg
I think the captain of the Titanic believed he could actually avoid collision altogether, which is another scenario. Perhaps the captain felt that if he hit an iceberg at all, his career would be finished, and that he might as well risk going down with the ship if it gave him any chance to escape collision. Some parallels there in project management!

James Bullock
So, we’ve got some kind of balancing between multiple kinds of consequences. And if that’s done during the event, it slows down decisions even more. I think that’s one of the ways that shared values and "alignment" are so powerful. People have an agreement on what is important, before they have to react in real-time.

Dan Starr
This prompts me to think about the difference between redundant and nonredundant systems. If you have multiple processors, file servers, communications hubs, whatever, the appropriate response to a fault condition may be to go ahead and take a chunk of your system down, letting the redundant parts of it keep your operation "afloat." If you have but a single system, you don’t have that option–your only approach to keeping your system operating is to avoid failure conditions and to accept that when they occur, the results will be catastrophic.

Another observation: At first glance, the idea that the Titanic went to sea with a woefully inadequate set of lifeboats would seem almost criminal. But think about it more, and it seems analogous to the situation we have today with air travel. You don’t get a parachute (or a glider) when you get on a commercial airliner, and the basic assumption is that if the plane falls out of the air, all aboard will probably die. Instead of providing escape systems, the airline industry tries to minimize the chance of having an in-flight disaster. This is the same as the model I think was behind the Titanic’s "inadequate" lifeboats–if the ship sinks, all aboard will die, so we must minimize the chance that the ship will sink.

Jerry Weinberg
It could be that they thought there were adequate lifeboats for a different scenario. In that scenario, the lifeboats are there to relay people to another ship in case something happens that cripples the Titanic. After all, they believed the Titanic was unsinkable, so in case of a crippling accident (the worst scenario they imagined), you simply wait on board until the rescue ship arrives and then transport your passengers to safety. I was on the QE2 once when we prepared to do this (in response to a bomb threat). Another design principle: If you can serialize, you don’t need so much instantaneous capacity. If you have a software development project (as opposed to an operational situation) that requires you to make some decision faster than overnight, that project has already failed.