A textbook case: next chapter

Update: A textbook case

In a previous posting I made the claim that the problems with the rollout of Healthcare.gov were a text book example of problems Brooks describes in the Mythical Man-Month. If misery loves company, and if you are a NASA software developer, the continuing reports about Healthcare.gov should provide reassuring proof you are not alone.

The Washington Post reported today that, last spring, the administration was warned by 'independent' consultants that the website deployment might be delayed. A number of risks were listed, but the following was especially telling:

...the policy and requirements of a program are best defined at the outset, leaving sufficient time for testing and revision. By contrast...the federal marketplace’s design was marked by “evolving requirements” that shifted throughout the design phase, leaving scant time to test the system before its launch.
(from Private consultants warned of risks before HealthCare.gov’s Oct. 1 launch, November 19, 2013 Edition of the Washington Post)

For many of us in the software development business, the report of requirement creep will have a creepily familiar ring. Requirement creep is inevitable. It's always been with us; it always will be. (A topic for later posting.) Here is particularly trenchant example of requirement creep and the Humpty-Dumpty Effect¹ that occurred four decades ago during development of the Shuttle.

Even though NASA engineers estimated the size of the flight software to be smaller than that on Apollo, the ubiquitous functions of the Shuttle computers meant that no one group of engineers and no one company could do the software on its own. This increased the size of the task because of the communication necessary between the working groups. It also increased the complexity of a spacecraft already made complex by flight requirements and redundancy. Besides these realities, no one could foresee the final form that the software for this pioneering vehicle would take, even after years of development work had elapsed, since there continued to be both minor and major changes. NASA and its contractors made over 2,000 requirements changes between 1975 and the first flight in 1981. As a result, about $200 million was spent on software, as opposed to an initial estimate of $20 million. ²

The Post article hints at another area that mirrors the NASA development culture. According to the article³ "programs of this scale are ideally pursued in a more orderly process..." That phrase was lifted by the Post from a slide deck that was provided by the House Energy and Commerce Committee. The consultants deserve kudos for showing a particularly deft touch at reporting what appears to be a poorly implemented development process.⁴

Soft pedaling concerns has become a way of life for reviewers--especially reviewers whose livelihood depends are the next contract. This sort of conflict of interest is commonplace because there is small community of experts with the requisite skills to offer a meaningful opinion.

For those in positions of high authority, this is an old problem. After all, can you believe someone who has a vested interest in telling you what you want to hear? Machiavelli weighed in on this point. He had the following advise for Lorenzo de’Medici.

It is an infallible rule that a prince who is no wise himself cannot be well advised...The counselors will all think of their own interests, and he will be unable to either to correct or to understand them. And it cannot be otherwise, for men will always be false to you unless they are compelled by necessity to be true. Therefore it must be concluded that wise counsels, from whoever they come, must necessarily be due to the prudence of the prince...⁵ [my emphasis]

In other words, Machiavelli suggests that for effective governance, the management must be wise about the subject at hand and receive true counsel from her advisors. Ironically, successful management of a government technology project depends heavily on the one's ability to craft the acceptable message. So much the better if the message is true.

^1. See The Humpty-Dumpty Effect for a discussion what happens when work is divided between engineering teams.
^2. Tomayko, J., Computers in Spaceflight: The NASA Experience, Chapter Four-Computers in the Space Shuttle Avionics System-Developing software for the space shuttle. NASA contractor report, 1988. Page 114.
^3. ibid
^4. Given the reported lack of coordination between development groups, this is anything but surprising.
^5. Michiavelli, N., The Prince. Translated by Ricci, L. Revised by Vincent, E.R.P., Oxford University Press, World Classics. Reprinted 1968. p.108.

INEDIA, the new, improved FBC

From Ratus rattus: A digression from the previous post

Excerpt
"Projects overrun because the NASA business model encourages under bidding and the promise of unrealistic expectations."

World Peace, World Hunger and a World Series win for the Astros were overlooked

In the last posting, I listed a few "frustrated utterances of immutable facts of life at NASA" that adversely impact the lives of the software development community. Starting with this posting, I'll launch into a perfectly foolish effort to dispute a few of these "immutable" facts with the hope of describing a Agency culture where it's rational to develop the kind of software needed for the next-generation space system.

I'll start with frustrated utterance #1: NASA's business model encourages underbidding and over optimism. Why is that? Bear with me for a few mental switch backs.

First stop, the money. For the last decade NASA budget has hovered around $17B per year.¹ While this is a lot of money, it's small potatoes in the world of government programs. Here's a few comparisons:

• Homeland Security has had a budget of roughly $55B since 2011. That includes $8B for TSA in 2012. That's nearly twice the size of NASA 'Science' program.^{2 7}
• The Farm subsidies budget has varied from %15-24B since 2000. The subsides are controversial since much of the money goes to large corporate farms.³
• The 'Prisons and Detention' budget has around $8B since 2011. A needed expediture, but an uncomfortable comparison with the NASA science budget.⁴

No criticism of these programs is intended. I only mean to provide some reference for understanding that, in the grand scheme of things, NASA's budget is modest. On the other hand, NASA stated aspirations are anything but modest.

What about the NASA budget itself? It's not what it used to be. The degree of decline might surprise you. Consider the following:

• In the years from 2000 to 2013, NASA budget remained flat at around $15-17B. ⁵
• In 2004, the President announced a major space initiative to return to the Moon and Mars. There was no matching budget increase to accommodate this major new program.⁶
• During the Apollo program, NASA's budget was over 4% of the Federal Budget
• From 1975-2000, the NASA budget represented about 1% of the Federal Budget
• The 2013 NASA budget is about 0.5% of the Federal Budget.

That $17B has to go a long way. It supports all the programmatic goals called out in the Mission Statement. About 30%, or $5B, is allocated to 'Science'. About 60% is allocated to 'Exploration'⁷ (for developing and operating the shuttle replacement systems). The rest goes to odds and ends.⁸ Once you throw in all the piece parts of the NASA mission statement, it's easy to conclude that the Agency menu calls for steak on a hamburger budget. How can Headquarter get by with that? Cleverness counts.

Two decades ago, the Administrator introduced a new approach for NASA missions, "Faster, Better, Cheaper (FBC)". In theory, the work could be streamlined; the funding gap would be made up by efficiency. At first, the policy looked like a winner. The first FBC mission Mars Pathfinder was a huge success. FBC was anointed as the new NASA business model.

However, the Pathfinder success wasn't what it seemed. According to a manager who played a key role on that project, Pathfinder met it's budget because it received significant help from institutional investments and technologies contributions from other missions. In other words, the actual cost of Pathfinder was really significantly higher than the FBC budget. It just appeared to be less because the project managers were good horse traders.

Subsequent missions would not enjoy the same advantage. The result: two mission failures, a slew of cost overruns, an over worked and dispirited engineering staff and a halt in the infusion of new mission technology. Almost everyone I know considers FBC a sham. The standard retort around the Agency was, "Faster, Better, Cheaper--pick two."

FBC endures as a set of cost-caped science programs.⁹ Unfortunately, due to the mission failures, the intended efficiencies did not survive and the Agency is now rife with oversight policies and processes that drive up cost. As if that wasn't enough, labor costs have increased sharply in the last decade. In other words, costs are rising, budgets are declining and expectations remain unchanged.

Perhaps the policy that underlies the NASA business model should really be called, "It's Not Enough, Do It Anyway (INEDIA)."

The realities of INEDIA require that something must give. But what? In order to understand how the sacrifices are chosen, it is necessary to understand NASA's real role in the government. In other words, who's the real NASA customer and what do they expect in return for the $17B?

Many would say that the science community is NASA's customer. In my unorthodox view, the real customer is sitting in the White House. NASA provides each President with a lasting legacy of concern for the future and ensures our national prestige as the leader among nations. The scientists are a really just a focus group. In other words, science is not primary; it's a means to an end. The value that NASA provides any administration is as public relations generator--$17B worth of public relations. If science generates a public-relations benefit, then the customer is satisfied.

If you except this assertion, here's a few conclusions that follow:

• Myths are good PR. NASA is in the business of science myth-making on a fixed budget. Rover tweets are cheap. Cool webpages with galactic photos and bios of glamorous young engineers galvanize the nerdy imagination.
• Science proposals must have a gee-whiz quotient with a PR payoff.
• Science discoveries have a short life in the news cycle. Space dramas like the MSL landing, the MER Landing or the final Hubble Repair Mission focus national attention. The important scientific discovery that there was once a lake on Mars had a half-life equal to last week's Monday Night Football game.
• The PR beast must be constantly fed--doesn't matter if it's only a celebration of the trivial.
• Science programs receive a modest proportion of the budget because the manned program tends to provide better space drama. This is captured in the oft repeated aphorism "No Buck Rogers, no bucks."
• NASA can poke along with it's programs so long as there's no disasters. Mission failures are bad PR. The President is too busy with real crises and doesn't need one from NASA.
• The lack of space drama has created a crisis of direction in the Agency.

In other words INEDIA, but don't foul up.

---

So what's a NASA manager to do? So many areas in the mission statement, so few funds. Hard decisions like the closure of a NASA center or the abandonment of program are fraught with political peril. But, worry not. A system has evolved that preserves all the hobby horses. Here's how it works:

At this level of planning, it doesn't matter if the 'e' comes before the 'a' in software

Most of the sound and fury illustrated in the diagram is dedicated to the politics of consensus. Consensus is essential. No one wants a nattering nabob disrupting the harmony.

The engineering, the underlying reality, shows up in the upper right around the "proposal development" box. That's where the politics and engineering get mixed in a recipe that looks something like this: Stir in the need for gee-whiz. Add a reduction of development managers with an extract of dubious commitment justified by facile assumptions of reuse and inheritance. Whip up a fluff of directives from upper management who fret about keeping an institution open. Viola! There you have it. An INEDIA-faithful mission proposal infused with elevated expectations and a budget/schedule time bomb. As the wags back at work used to say, "The Director never met a proposal he didn't like."

If you happen to be a software manager who added ingredients into the brew, you would now be responsible for flight products based on inherited software. You would be under budget pressure because your budget is too large. No one on your team believes that the as-built, inherited system will be deployed because it never is. The works starts; the schedule gets chewed up. In time the actual requirements appear. Before long it becomes obvious that schedule and budget relief are needed. But a software manager has no authority. Your project management dismisses your concerns and assumes these are merely the usual scare tactics of extortion. The proposal commitments are sacrosanct. A project manager's reputation depends on devotion to the proposal cannon. Once again, it's time to apply the screws to those undisciplined programmers. But, as the launch date approaches, the truth of testing cannot be repressed. Project management finally orders up reserves from the keep. By then it's late in the schedule and the team has no choice but to cut features, apply ad hoc patches, and take design short cuts. And so, another task slips budget and schedule.

Perhaps the worst of this tired tale is that the resulting spaghetti-like architecture is destined to become the basis of the next mission proposal and the cycle repeats. It's a Brooksian Tarpit.

---

How might things change? Is a INEDIA business model for NASA simply the intractable result of the natural forces alive in a government bureaucracy. Could it be different?

I firmly believe so. If for no other reason than once the Chinese or Indian Space programs land systems on the Moon and Mars, there will likely be mounting political pressure to reassert the Nation's leadership in space technology. And, it's not just the Chinese and Indians; Brazil, Argentina and Iran are emerging nations who have their own space program. If that doesn't turn the tide, an asteroid scare could change the equation. Eventually, an administration will be called on to address the loss of national prestige. When that happens, the Agency goals will be refactored so that budgets match expectations.

It need not come to that. Here's a few reckless suggestions for how the Agency might escape the shackles of the INEDIA cycle.

• Scale back the NASA Mission statement to match the budget
• Assume that the most exciting mission will be enabled by engineering improvements. In particular improvements to Systems and Software Engineering that address that complexities of building a software-intensive space system.
• Add a focus group who's objective is to define projects and programs that generate space drama and national prestige.
• Alter the proposal process to address the full cost of ownership and prevent low balling the development costs
• Reward management that can accurately sniff out the false cost assumptions claimed from reuse and inheritance.
• Establish a 5-year suspension for project proposers who overrun budgets and slip schedules because of low-balling development and operations.

I know it's crazy, but the current way of doing business is not immutable. It could change. A new business model might just revitalize the Agency... even for a paltry $17B.


^1. See slide 7 of NASA 2013 Budget Estimates
^2. Homeland security Budget p.25
^3. Farm Subsidy Table on Environmental Working Group (EWG) web site
^4. U.S. Department of Justice Overview page 6.
^5. NASA Annual Budgets
^6. Constellation Program
^7. Exploration program has officially replaced 'manned' program because of the gender bias implied in the name.
^8. By comparison, the 2013 European Space Agency Budget (ESA) budget is about $4B. ESA allocates only 9% to the human space flight program and the rest developing and flying pure science missions. In other words, the NASA and ESA science budgets are roughly equivalent.
^9. For example, the cost cap for a recent Explorer Program Announcement of Opportunity (AO) was $30-60M. The Discovery Program missions have a cap of $425M. The last New Frontiers AO (2009) had a cost cap of $650M. This may seem like a lot of money, but compared with 2.5B for MSL, $9B for the James Web Telescope, and $450M per Shuttle Launch (not including the large recurring infrastructure costs to keep JSC and KSC operational), these budgets are not generous.