Risk Avoidance and other Eggistential Consequences

From Chapter 2: The Mythical Man-Month 

Omelet served at Sunrise High Sierra Camp in the summer of 2007

Excerpt
An omelet, promised in two minutes, may appear to be progressing nicely. But when it has not set in two minutes, the [Antoine's] customer has two choices—
wait or eat it raw. (Page 21)

From "The Last Lion, William Spencer Churchill, Visions of Glory, 1874-1932"¹

Excerpt
"We simply couldn't have failed...and because we didn't try, another million lives were thrown away and the war went on for another three years. (Commodore Roger Keyes, British Royal Navy. Page 542)"

From "Lee's Lieutenants, A Study in Command, Volume III"²

Excerpt
Our Corps Commander was simply waiting for orders when every moment of time could not have been balanced with gold.
(Captain James Power Smith, CSA, reporting on a meeting of Confederate Generals. About 4:00p, Day 1. Battle of Gettysburg, Page 94)

Brooks' observation that an omelet takes time to set implies a challenge question: Why would the omelet still be raw when the chef promised it would be done? Brooks might explain with a quote from Yogi Berra, "It's tough to make predictions, especially about the future."

Brook's suggests a remedy: provide managers with better schedule estimation tools. The hope is that a calculus of schedule estimation would produce accurate schedules just like an egg timer produces 'just right' eggs. With better estimation tools, managers could "stiffen their backbones," defend realistic schedules and dispense with those "patron-appeasing," "gutless estimations" that lead to disasters.

I'm skeptical.³ By way of a brief refresher, NASA is a very large, hardware-centric, government-sponsored, bureaucracy responsible for high-profile demonstrations of an administration's commitment to tomorrow. The more space drama, the better. That changes the rules. So long as the government forks over funds, there is no real schedule disaster. The rules for projects living off government funding are quite different than those that depend on profit.

With that in mind, here are a few alternative suggestions for why the omelet would be raw at eating time:

• The customer expected Antoine's to provide the same service as McDonald's because he considers the omelets equivalent.
• The chef spent the majority of the allotted cooking time asking all his colleagues if they approve of his cooking oil.
• Obtaining collegial consensus about cooking oil always takes longer than anticipated.
• The chef is willing to serve a raw egg because he knows that if the omelet is sent back, he'll have plenty of time to set things straight later.
• The chef knows the customer simply craves the moral superiority that comes from eating at Antoine's. The omelet could be slimy as a Louisiana lagoon or as chewy as a tire. Doesn't matter; why worry?

In each instance, the outcome depends on how the chef spends his time. So it is with software projects.

While I worked at NASA, we were always on a tight schedule. Nevertheless, we spend huge blocks of budget and schedule on content-starved reviews that necessitated plenty of handwringing but seldom resulted in a decision of consequence. I can't recall an instance where a bold decision was made to grab an opportunity. So while we sat around conference tables listening to monotonous PowerPoint talks, grand displays of reviewer intellect and solemn rehearsals of managerial discretion, opportunities sped past us like stripes on the interstate. It was apostasy to suggest we were wasting our resources and that we could have done better.

I have a (somewhat perverse) interest in lost opportunities that changed the course of history. The catalog of military history is ripe with these stories. A couple of mostly-forgotten examples come to mind: The British catastrophe at the Gallipoli and a lost opportunity at Gettysburg.

Making of Gallipoli

The failure of British leadership at Gallipoli unfolded with the inevitability of a Shakespearean tragedy.⁴

On the Ides of March, 1915, Rear Admiral John De Roebeck assumed command of the Allied Fleet in the Mediterranean. His assignment was to force the Dardanelles, cross the Sea of Marmara and take Istanbul. If successful, the Turkish regime would collapse and expose the Kaiser's armies to a devastating attack that would end the war.

On March 18, a De Robeck ordered the attack. The fleet sailed unopposed into the Dardanelles and proceeded to decimate Turkish defenses. Just as the ships were entering the Sea of Marmara, the French battleship Bouvet hit a mine. Moments later three more battleships hit mines. Fearing that the Turks were floating mines towards his fleet, De Robeck ordered a general retreat.

The Turks had done no such thing. The De Robeck's ships has merely strayed to close to the south shore and stumbled into a haphazardly prepared minefield. In fact, the Turks were beaten. The government was preparing to abandon Istanbul; the country's gold and art treasures had already been removed.

But the British were cautious. The top brass decided that the naval effort was inadequate and needed to be supplemented by an infantry assault of the Gallipoli peninsula. What followed was an ill-fated invasion that was caused by a failure of British leadership that Liddle Hart described as "a chain of errors in execution almost unrivalled in British history."

General Ian Hamilton was given command of an Allied expeditionary force ordered to capture Gallipoli. He sailed ahead of the troops to begin planning with De Robeck. Shortly after arriving, Hamilton conducted a cursory survey of the Gallipolian terrain and concluded that three weeks would be needed to prepare for the assault. One delay led to another. The British would not land until April 25th, a 5-week delay. In that interval, the Turks, with help from the Germans, prepared a strong defense. The subsequent British-led invasion lasted nearly 8 months. Allied casualties amounted to nearly 200,000 with over 50,000 killed in action. Turk casualties were comparable.

If only De Robeck had ordered a follow-up attack on March 19th, there would have been no debacle at Gallipoli and a German surrender might have come in short order. Instead, the war would continue for another 3 years.

Ewell at Gettysburg

The Battle at Gettysburg was the turning point in the Civil War. The Confederates might have won Gettysburg if Lieutenant General Richard Ewell⁵, the Commander 2nd Corps, had taken the initiative.

On July 1, 1863, the Union and Confederate Armies collided at Gettysburg. About 4:00 pm, Ewell's Corps drove the Federals south through the town onto Cemetery Ridge. The Yankees were vulnerable as they scrambled to consolidate their hold on those heights. A moment of victory was at hand.

A brigade commander, Brigadier General John Gordon saw there was a magnificent opportunity to rout the Yankees. Gordon approached Ewell to obtain the order to charge the enemy. No orders were forthcoming. Ewell was frozen. "Inwardly, something had happened to the will Richard Ewell."⁶

A short time later, Brigadier General Isaac Trimble rode up. Trimble recognized that the Federals had not yet occupied the heights at either Culp's Hill or Cemetery Hill. Trimble was one oldest Rebel officers and was known for his lack of tact. According to tradition, Trimble said to Ewell, "We've had a grand success; are you not going to follow it up and push our advantage?"⁷ Ewell replied that he needed further orders from General Lee. Trimble persisted, "...give me a Brigade and I will engage to take that hill." Ewell responded with silence. In an act of insubordination Trimble then said, "give me a good regiment and I will do it." When Ewell still refused to answer, Trimble gave up.

By next day the Federals had dug in on both Cemetery Hill and Culp's Hill. The Confederates would stage a day-long series of intense attacks, but they never secured those heights. Ewell's failure to act had allowed the Yankees to secure Cemetery Ridge where, two days later, they would beat back Pickett's charge and defeat Lee's Army.

A senior manager once told me, "We are a risk adverse institution. We do not break the rules." Another senior manager repeatedly scolded me that "Things happen slowly here. We change by evolution, not revolution." From a bureaucratic perspective, this advice was impeccable--after all "Failure is not an option."

Being a reader of history, I was painfully aware of the implications of indecisive or overly cautious leadership. While the over caution that characterizes NASA management does not cost the tens of thousands of lives, it does result in the waste of tens, if not hundreds, of millions of dollars. If "commercial" contractors like SpaceX or Orbital have an advantage over NASA it's simply that they are willing to take reasonable risks.

Excessive caution was the bane of most of my forward-thinking colleagues who saw the few available opportunities awarded tasks that were the innovative equivalent of putting new paint on an old car. But that's not the worst of it. I believe the true concern was best captured by a frustrated system engineer with the following epigram: "The trouble with evolutionary change is that relies on extinction."

^1. Manchester, W. "The Last Lion, William Spencer Churchill, Visions of Glory, 1874-1932. Laurel Trade Paperback. 1983.
^2. Freeman, D.S., "Lee's Lieutenants, A Study in Command, Volume III". Scribner and Sons. 1944.
^3. See my previous post, In the beginning... there is the estimate.
^4. Churchill's role in the Dardanelles has stirred plenty of controversy among historians. Manchester lays the disaster at the feet of De Robeck and Hamilton. On the other hand, Correlli Barnett in Engage the Enemy More Closely: The Royal Navy in the Second World War (Norton & Co. 1991), lays most of responsibility on Churchill and points to a similar 1940 disaster in Norway. Both accounts are extremely well researched and convincing.
^5. Ewell had replaced Stonewall Jackson who died from wounds received at the Battle of Chancellorsville in May.
^6. Freeman. p.92.
^7. Freeman. p.94.