The peculiar arrangement

From Chapter 3: The Surgical Team

Excerpt
In the surgical team, there are no differences of interest, and differences of judgment are settled by the surgeon unilaterally. These two differences—lack of division of the problem and the superior-subordinate relationship—make it possible for the surgical team to act uno animo. (p. 35)

It's summer; at least for all practical purposes. My attention has drifted north to the trails, cirques and passes of Sierra's. The thought of alpine air and mountain vistas reminds me of the exalting promise I once felt when developing software for space applications before realizing that the actual work takes place behind communication barriers in a bureaucratic miasma. So today, my thoughts have returned to Brooks' Law and the Mythical Man-month.

Brooks has a solution for the communication problem: the "Surgical Team." (See "A fly in the ointment" for a description of the communication problem described by Brooks' Law.)


If you've been reading along, you'll remember there are just two technical contributors on the 10-person "Surgical Team": The "Surgeon and the "Co-pilot." (If not, see "Picking the Archetypes for the Surgical Team".) The Surgeon is the chief. The Co-pilot is #2. The Surgeon's "alter ego." As #2, he or she shares the same responsibilities and skills as the Chief. However #2's responsibility is restricted to that of a sidekick, "thinker discussant" who knows all the code and, maybe even, writes some.

This strikes me as a peculiar arrangement. What exactly is #2 authorized to do on the Surgical Team? In Brooks' scheme: nothing.

For example, what if #2 meanders over to the Editor's cubicle and drops off a few changes to an ICD (i.e. Interface Control Document). Would it be prudent for the Editor to make alterations without being confident they meet the Chief's approval?   Hardly.

The co-pilot is only supposed to tap into team communications
From Mythical Man-Month p. 38 (1975 version)

And what if the Editor has questions? Should she rest easy that #2 has the authoritative answers? Only at her own hazard. And, possibly, the task's since without a clear line of technical authority there is a risk that each team member will develop to an independent concept of the interface. Same goes for the Administrator, Programming Clerk and the rest of crew.

So, in fact, if #2 talks to anyone, his presence probably increases the number of communication paths by N-squared and defeats the very intent of the Surgical Team. That was certainly my experience—I can't recall a time when a reliable decision came out of an audience with a #2.

So, what could reasonably expected from #2?

It's a relevant question for NASA.  There is a pandemic of #2's. Typically, they go by names like "Deputy," "Assistant," or "Associate." There are "Deputy Section Managers", "Deputy Architects" and "Assistant Division Managers." That's not to mention "Deputy Division Chiefs", "Assistant Project Managers", "Deputy Directors", and "Associate Administrators". There are even "Deputy Associates" and "Associate Deputies." The list goes on. Any Chief with a scintilla of actual authority is likely to get a #2.

Some chiefs have limited authority

However, the provisioning of authority does not always rest with the Chief. In fact the relationship between the Chief and his #2 may be the best indicator of the Chief's standing in the woof and warp of power and organizational influence.

For example, a chief may get stuck with a #2 whose real job is to guard the interests of a higher authority and ensure the chief steers the party line. Or, he may be stuck with a fallen  manager who can no longer land a portfolio and must be parked in a desk job. Or, worse, he may be assigned an-anointed-young-turk who carpools with an executive VP.

On the the other hand, the chief may be a "made man" and have actually authority. In this case he may select his own #2 and use him as he sees fit. Here's a few examples:

Some chiefs have clout
(Meyer Lansky did)

Attack dog
The best defense is a good offense, preferably one that preserves the appearance of impartiality. #2 can keep anyone with an opposing view safely on the defensive. For example, just as a Vice Presidential candidate is expected to relentlessly hector the opposition, a worthy #2 will undercut technical initiatives that threaten the Chief.
Henchman
Every job has its dirty deeds. For example, since the Agency does not tolerate failure (see "Failure is not an option"), remedial measures, like blame displacement, are frequently needed. #2 can do that dirty work so that the Chief's hands remain bloodless.
Parade sweeper
Leadership is not merely a matter of opining with one's feet on the desk — especially in a a paperwork hungry bureaucracy. Someone must take on the stultifying job of feeding the beast meaningless paper. Here's where #2 may make his most meaningful contribution.
Spy
Knowledge is power. A smart Chief must know what the troops are thinking. But, a Chief can have no real friends; unfiltered information seldom penetrates the management bubble. There's too much to be gained by flattery and it's too tempting to rail against leadership. #2 can help. She can be a friend, a trusted advocate for the working stiff, and a reliable reporter of the sentiments that lie beneath the surface.
Hand holder
Two heads is better than one; at least when it comes propping up the boss. With #2 at his side, a Chief makes a much more commanding presence. Best of all the Chief can be assured of close support should he be faced with dissent. There's no better hedge against those unsettling moments of insecurity.
Note to Chief: Effective hand-holding requires occasional dissention to preserve the appearance of honest and considered support. Don't be alarmed.

---

Gaius Gracchus, tribune of the people,
presiding over the Plebeian Council

A short digression
It was different once. Back in the Roman Republic¹ Tribunes² could only exercise their authority in person.

Tribunes were the elected representatives of the plebeians (i.e. non-aristocrats that held property). They were charged with protecting plebian interests. However, a Tribune was not a magistrate. A Tribune could not create laws, render judicial decisions or have any of the other powers of a government official.

Rather, Tribunes had the peculiar power of being able to veto any magisterial decision, including decisions made by Rome's highest elected official, the Consul. No one could stop a Tribune. But there was a rub. The Tribune had to physically present while the act was occurring. For example if Marius ordered the execution a political opponent, a Tribune could step in and stop the process. But the intervention had to be in person. No deputy. No Associate Administrator. No house guards. No #2 of any label. Only the Tribune's physical presence would do!

In recognition of the threat to life and limb, Tribunes were sacrosanct. Sacrosanctity meant that no one was allowed to touch a Tribune. Any interference was punishable by death. But there was a rub. The minute the tribune left, the act could be completed. The veto had no legally lasting effect.

Imagine how that might change things today.

---

And then there's the politically motivated selection of #2. This was the case when Napoleon appointed General Jean Victor Marie Moreau to command his Rhine Army.

General Jean Victor Moreau
(1763-1813)

Moreau was a popular member of the French aristocracy and a distinguished general of proven merit. In 1799, the Directory removed Moreau from his command—Moreau's loyalty had been unfairly questioned. It was a period of political instability and guillotine-fueled paranoia. But, the French armies were suffering defeats in the Austrian war; the Directory was desperate for talented field commanders. Moreau was re-appointed, but ill feelings and mutual distrust persisted.

Enter Napoleon. On his return from Egypt in 1799, Napoleon staged a succesful coup d'état that overthrew the Directory.³ Moreau played a key role. As a reward Napoleon named Moreau commander of the Army of the Rhine which lent credibility to Napoleon's fledging reign as First Consul.⁴

The good feelings would not last. Just months after his receiving his commission, Moreau refused to execute Napoleon's brilliant plan to invade Italy by crossing the Alps.⁵ Moreau told Napoleon that he should "follow the routes of earlier campaigns." Bad mistake. Napoleon would later write, "It was impossible to overcome the obstinacy of Moreau...He at first refused to command under me...he objected to my plans, pretending that passage of Schaffhausen was dangerous." For the moment, Napoleon did nothing in response to Moreau's insubordination. According to Napoleon, "I was not yet sufficiently firm in my position to come to an open rupture."

Three years later, Napoleon would find the opportunity to dispose of Moreau, but associating him with a Royalist plot. Moreau was forced to flee to the United States, but returned 10 years later to help the Swedes and Russians defeat Napoleon.⁶

Note to Chief: Don't forget that #2 might reappear as part of antithetical cause. Be nice or be merciless.

Louis Alexandre Berthier
(1753-1815)

The tale of Moreau contrasts sharply with Louis Alexandre Berthier. Berthier was also a highly respected member of the aristocracy who served with Napoleon from the initial campaigns in Italy till the 1813 defeat. Berthier was brilliant, a quick study of military necessity and a master of detail. He was by far, Napoleon's most able staff member. No one was surpassed him for understanding and executing the Emperor's commands.

After Napoleon was shipped off to Elba (1814), Berthier retired. When Napoleon returned for the 100 days, Berthier was refused to rejoin the Grande Armee. Two weeks before the Battle of Waterloo, Berthier died by fall from a window.^7,8

Note to #2: Your fate may depend on your chief. Enlist with care.

---

Being #2 is no picnic. For some it's a rite of passage. For others a last resort. But, whether a deputy is motivated by ambition or self preservation, he is not his own master. So observe your #2. Read the tea leaves. There's a lot that can learned about the real politics of your organization. If you aspire to be Chief, your future may depend on it.

Speaking of the future, I see an indigo sky and a few Sierra mountain passes up ahead.

---

^1. The Roman Republic is traditionally dates from ~500-27 BCE when August founded the Empire.
^2. Tribune is derived from tribe.
^3. Napoleon's coup was a close call that required a good bit of intrigue.
^4. In the beginning Napoleon shared power with two other men: Jean-Jacques-Régis de Cambacérès and Charles-François Lebrun. Within a year, there was a new constitution that consolidated all power in hands of the First Consul. In 1804, Napoleon would declare himself Emperor.
^5. The strategy would lead to the victory at Marengo, one of Napoleon's greatest achievments.
^6. He died at the Battle of Dresden.
^7. Many historians surmise that Berthier was assassinated. Throwing someone out a window is called Defenestration and is a time honored form of execution.
^8. Chandler's "The Campaigns of Napoleon was the source for these anecdotes. Chandler, D. "The Campaigns of Napoleon. The Mind and Method of History's Greatest Soldier." Scribner. 1966. p. 269, 308.

Buyer's guide for the NASA software catalog

From NASA's Software Catalog

Excerpt from catalog cover letter
From the rudimentary but effective Apollo Guidance and Navigation System that landed the first humans on the lunar landscape to the 500,000 lines of code used to put the Mars Curiosity Rover on the surface of the Red Planet, software has always been at the core of NASA’s mission successes…The technologies featured in this catalog represent NASA’s best solutions to a wide array of complex problems, and they are on offer here to the public for use.
—David Lockney, the Technology Transfer Program Executive

caveat emptor

Last month the NASA Technology Transfer Program announced its new Software Catalog. It's a listing of NASA software that's available for the asking.

This is an impressive list. According to the Press Release (14-102), "NASA is making available to the public, at no cost, more than 1,000 codes ..."

I am not sure what's meant by "codes." Could they mean source code? Could they mean libraries? Could they mean executables? One thing is certain, it's unlikely the release was written by someone who doesn't know much about software. I digress.

The catalog lists over 160 pages of software products. The products are organized into 15 categories including "Data Servers Processing and Handling," "Crew and life support," "Autonomous Systems," and "Vehicle Management" just to name a few. Some of the items look pretty interesting. Here's a sampling:

• Station/Orbiter Multibody Berthing/Docking Analysis Tool (SOMBAT).
  U.S. Government Purpose Release, Page 109
  Described as a "multibody dynamics and control system simulation tool." If this is anything like the DARTS/DSHELLproduct developed at JPL, there must be a lot of math in this package. I wonder if SOMBAT was ever used for flight. I wish I'd been at the team meetings where serious-minded engineers were gravely proclaiming the importance of SOMBAT. It's a perfect subject for Vi Hart. She sings about a Laserbat. Could there be a connection? (I strongly recommend viewing Vi Hart's marvelous Twelve Tones video.
• Station Spacewalk Game App
  General Public Release, Page 109
  Described as simulations of Extravehicular Activities (EVAs) conducted by NASA astronauts on missions to the International Space Station. I really wanted to try this one out. To my disappointment, the game crashed two different browsers. Bummer.
• Real-Time Kidney Stone Tracking Algorithm
  U.S. Government Purpose Release, Page 116
  Described an algorithm that uses focused ultrasound to clear stones from a kidney. The software is a component of the "Rolling Stones prototype;" i.e. you need the Rolling Stones software. (Not in the catalog.) I'd give this offering high marks for its low pun.
• Rover Software (RoverSW), Version 1
  Open Source Release, Page 116
  Described as a modular, extensible framework for exploration robots. The core provides the for building a service-oriented architecture that uses middleware. Hard to believe this code has flown because 1) it's open source¹ and 2) no current flight system, I know of, would waste the onboard cycles on middleware. One day. Still, I bet it is a fun research project and might be just be the thing for a graduate student project.
• MAVEN Flight and Ground Software
  U.S. Government Purpose Release, Page 116
  Described as a flight/ground package. Here's some substance. The products include simulation models, regression tests and even the flight code that runs on the spacecraft's flight processor. This is a proprietary product of Lockheed Martin Space Systems that's built on the technology developed for JUNO and MRO. It may not be the latest technology, but it's the real deal. I wonder what would happen if SpaceX wanted to license it.

Notice the release categories. Every item in the catalog has been assigned a release category. The categories define access requirements and restrictions. If you want some of this "no cost" software these constraints on release are important.

Here's the description of the release categories from from the catalog:

• General Public Release—For codes with a broad release and no nondisclosure or export control restrictions
• Open Source Release—For collaborative efforts in which programmers improve upon codes originally developed by NASA and share the changes
• U.S. Release Only—For codes available to U.S. persons only, with no further transfer of the software allowed without the prior written approval of NASA
• U.S. and Foreign Release—For codes that are available to U.S. persons and (under special circumstances)persons outside of the U.S.
• U.S. Government Purpose Release—For codes that are to be used on behalf of the U.S. government
• Project Release—For use under a contract, grant, or agreement
• Interagency Release—For use by U.S. government agencies
• NASA Release—For use only by NASA personnel and contractors

A quick search of the catalog produced the following breakdown of categories:²

• Items categorized as General Public Release: 27 (3%)
• Items categorized as Open Source Release: 111 (11%)
• Items categorized as U.S. Release Only: 213 (22%)
• Items categorized as U.S. and Foreign Release: 22 (2%)
• Items categorized as U.S. Government Purpose Release: 590 (61%)
• Combined total of the above: 963

In other words, nearly 60% of the offerings are available to the "public" only when they are working on a government contract. Makes me wonder what's meant by "public." Is it my imagination, or does it seem that the promise of "making available to the public, at no cost, more than 1,000 codes ..." is an oversell?

NASA Press Release (14-102)

It's worth noting that under Title 37, any software developed with government funds is available for free, under a Government Use License, to any business or institution doing government business.³ After all, the government doesn't want to pay to develop the same product twice.

More interesting perhaps is the NASA software that is not listed in the catalog. The excluded software should also be available under a government use license. Take for example, the software on the Mars Opportunity Rover; it is not in the catalog. Isn't that ironic since, the Opportunity software is featured in the catalog's cover letter written by the Technology Transfer Executive?⁴ (see the excerpt at the top of this posting.)

So let's say you find some software in the catalog that you want. Here's a few things to bear in mind:

• This is not commercial software. The software has probably not been "packaged" for use by a 3rd-party user. i.e. there may not be sufficient information about installation or use. (Not for lack of professionalism on the part of the development team.)
• The software may not be immediately available. Someone may have to tar it up and do a bit of head scratching to decide what's in the package. Budgets are limited and the engineer who does the packaging may do the work on their own time.
• Many of the items are "research grade." The software is the product of talented domain experts who work with limited requirements and have little or no budget for independent testing.
• Technical support may be iffy. If you need support or bug fixes, you will probably have to sign a NASA Space Act Agreement in order to transfer funds to NASA to pay the engineers who would provide support. (No fun here.)
• Don't expect a dedicated programming team. NASA funding is quixotic. Engineers have to move from project to project and task to task to stay employed
• Your needs may not be a priority. First dibs on engineering time will always go to mission needs. Unless you have management connections, you may be paying to bring a new developer up to speed or the response may not be timely.
• Don't count on getting an exclusive license for government projects. The law mandates that NASA must provide the software to anyone doing government work.
• As a rule, software that's been on a mission has been subjected to much more testing that non-mission software. However, this is no guarantee of quality.
• Some of the items in the catalog are decades old. The code may be very brittle.

So, before deciding to bet you project on a free NASA product, here's a few questions you might ask:

• Is the code under maintenance? What is the annual maintenance budget? (NASA budgets are not secret)
• Is there customer support? Who do I need to contract with to get the service?
• What's the bug reporting and repair cycle? How do I figure the cost of bug repair?
• How often are new versions released? How are patches handled; are they coupled with system-wide releases?
• Are the original developers still available?
• Are there any restrictions on the licensing? Can the license be exclusive?
• Was the software used on a mission?
• Is there a replacement product in the works?
• What kind of documentation is available?
• Is there a user community?
• Are there export restrictions (ITAR or EAR) of any kind?

NASA has a charter obligation to see that its "products and processes...benefit the lives of ordinary Americans and the U.S. economy." This make sense. It's good idea to get the maximum benefit from a government investment.

However, is the oversell necessary? At times it seems that excellence in engineering been replaced by excellence in public relations. At the very least, the catalog reflects an ignorance about software. After all, software is not like a hardware piece-part that still has value after sitting on the shelf a few years. Software only maintains its value if there's a team actively working on the product. If funding for the product is cut, the product dies. If that filter were applied, the catalog might have a hundred items instead of a thousand.

Despite the fact that "software has always been at the core of NASA’s mission successes," NASA does not act like it is in the software business. If it did, the NASA culture would be radically different.

It could be otherwise you know.

---

^1. The International Tariff in Arms Regulations (ITAR), prohibits NASA from distribution flight software without careful review by the State Department. Foreign licensees of flight software must obtain a export license and make regular reports about protective measures to the state department. Not fun.
^2. The figures in the list were obtained by doing a case-sensitive search of the catalog for the text of each restriction type. e.g. a search of "Open Source" produced 112 instances including 1 instance in the text. The total may be low because the naming of the restrictions is inconsistent. e.g. not all items include the word "release" i.e. "General Public" may be used instead of "General Public Release." While not exact, the figures are indicative.
^3. See Title 37: Patents, Trademarks, and Copyrights PART 404—LICENSING OF GOVERNMENT-OWNED INVENTIONS §404.4 Authority to grant licenses. Note: No license is required when sharing occurs for work that being done under the same task order.
^4. Despite the Federal Law, the Curiosity software is closely held. I worked on one project where the MSL management would not permit sharing of the code with a development team doing work under the same task order. I addressed the reasons for this in a previous posting: Sharing is no panacea.

Sasquatch, El Dorado and Bug-free Software

Digression from the previous post: A fly in the ointment

From They Write the Right Stuff ¹

Excerpt
What['s]...remarkable is how well the software works. This software never crashes. It never needs to be re-booted. This software is bug-free. It is perfect, as perfect as human beings have achieved....

Once upon a time...
there was perfect software

...That's the culture: the on-board shuttle group produces grown-up software, and the way they do it is by being grown-ups. It may not be sexy, it may not be a coding ego-trip -- but it is the future of software.— Charles Fishman, Fast Company, 1996

From Software and the Challenge of Flight Control²

Excerpt
A mythology has arisen about the Shuttle software with claims being made about it being “perfect software” and “bug-free” or having “zero-defects,”all of which are untrue.—Nancy Levison, 2013

After reading the last posting, a friend, from my NASA days, sent me a link to a 1996 Fast Company article by Charles Fishman about the on-board Shuttle team at Johnson Space Center. The article, They Write the Right Stuff describes how the team managed their work. For the remainder of this posting, I'll refer to the article as simply the "...Write...Stuff."

My friend, who is very familiar with how Shuttle software was developed and maintained, was pointing out that the "...Write...Stuff" described the shuttle software team as having a 10-to-1 ratio of programmers to testers—roughly the same ratio proposed for Brooks' Surgical team.^{3, 3a} That's an exception. The NASA software teams I saw were typically staffed with a 1-to-1 ratio of programmers to a combination of system engineers and testers. However, if you included management and process oversight staff (i.e. those who do not contribute directly to developing the product), the ratio was more like 4 to 1.

I found the "...Write...Stuff" disturbingly wrong headed. It portrayed the Shuttle software development methodology as the wave of the future——the very culmination of software development as it should and will eventually be for all-time. Fishman is unapologetic this claim.

As the rest of the world struggles with the basics, the on-board shuttle group edges ever closer to perfect software. — Charles Fishman

Sure, the "...Write...Stuff" was written over 15 years ago. It's a feel-good piece. It's written by a journalist (i.e not a software engineer) for a for-profit publication whose sales depend on stirring the heart of your wistful science romantic who adores tales of shiny things that are technical.

So why take the piece seriously? Because like Sasquatch , or El Dorato, the best tall tales could be true. After all, who knows what's really beyond the campfire. A monster? A city of gold waiting for someone with the will to find it? When understanding is limited, the unlikely seems credible and impossible disprove. Who's to say a tall tale of software perfection is a fiction? Well-meaning people who haven't lived in the woods or traveled the West can be taken in—especially when, they want to believe.

Consider the plight of your typical techno-political bureaucratic manager. His efforts to reign in those software people fail repeatedly. He has endured scores of broken commitments. He now believes but a fraction of what he hears. He is flummoxed. He is living the misery of the 'software problem'⁴. He hears of a cure all. A new tool; a better process; a silver bullet; a perfect team building the perfect software. He is a captivated. He has a course of action; a fix is in reach. Shuffle a few budgets; levy a new approval process, demand a new document, an additional review, a few more extra process steps.⁵ Voilà! No more problems. What could be easier?

In other words, articles like the "...Write...Stuff" foster a management expectation that there is a quick fix. When these fixes are imposed on development teams pressing to meet a delivery, they can do real damage. Improvement is hard. It's expensive. It's disruptive. It's risky. Perfection is the stuff of stories.

Space Shuttle Main Engine hoisted into the A-2 Test Stand, Stennis Space Center  (1979)

An abridged history of shuttle software⁶

The on-board shuttle group worked on the code that controlled the Shuttle rocket engines. The "...Write...Stuff" portrays them as a highly regulated, strictly-managed group who worked regular hours. These constraints were necessary because software had to work or risk human life.

While that is true, the interesting thing about on-board software is how it came to be. No account of the team's success is complete without accounting for the quarter century of prior effort. To understand why that matters, the history of shuttle software might be instructive.

Space Shuttle Main Engine Controller (2005)
The controlled was attached directly to the engine body

In July 1971, NASA contracted Rocketdyne to build the Space Shuttle Main Engine (SSME). Each shuttle main engines had a pair of dedicated, redundant controllers that were attached directly to the engine. These controllers managed the low-level functions of the main engines like servo control, command data converter, sensor data transmission and fault response. In the initial implementation, these controllers were Honeywell HDC-601s and the software was written in assembler. By early 80's it was clear an update was needed. Rocketdyne updated the SSMe controllers to the Motorola 68000 and rewrote the software in C.

In March 1973, NASA contracted with IBM to build the Primary Avionics Software System (PASS). PASS was comprised of two parts the Flight Computer Operating System (FCOS) and the application software. FCOS handled engine sequencing, steering and redundancy management by providing the controlling function for the software built by Rocketdyne. The application software which included guidance, navigation and systems management. PASS is the software that is discussed in the "...Write...Stuff."

PASS ran five 'general purpose computers.' IBM selected the AP-101/S processor which was based on the same architecture as the IBM 360. Earlier versions of the AP-101/S had been flown on the B-52 and the B-1B. The processor used a variety of word sizes depending on the function. For example, instructions could be either 16 or 32 bits. Floating point could be 32, 40 or 64 bit words. The average speed for math operations was about half a MIP. Program state for all process was preserved in 64-bit status word that was updated with every instruction cycle. As if that wasn't complicated enough, the processor was capable of handling 61 interrupts at 20 priority levels while preserving real-time constraints.

The five AP-101/S computers were divided into two separate systems. A quad-redundant, fault tolerant 'primary' system, and a Backup Flight Control System (BFS). PASS controlled both. All the computers processed the same data. This meant all the primary computers had to stay in synch. Reliability and safety in the primary was based on a voting scheme that checked all the computers to ensure they were producing the same result. If one computer had a different result, it was isolated.

If the primary system failed, BFS kicked in. That meant the BFS also had to be synchronized with the primary. Synchronizing the primary and backup systems would proved to very difficult and lead to a major defect that delayed the initial shuttle launch. The synchronization problem is a fascinating story captured in John Garman's article, "The Bug heard 'Round the World." ⁷

Shuttle software loads.
(from Tomayko, Developing software for the space shuttle )

Each processor had access to about 100K words (36-bit words) of shared memory. Memory was addressed by using a 16-bit word plus an extension for the last 4-bits from the status word. Since the memory was limited, software loads had to be swapped out at different mission phases. The largest loads were about 100K words for Ascent and Entry. Swapping software loads while maintaining system state must have been a bit tricky.

HAL/S code snippet
from http://history.nasa.gov/computers/Appendix-II.html 

PASS was going to be (and still would be) a difficult programming job. After coding the Apollo software in assembler, the engineering team knew PASS would be too complicated to write in assembler. After much debate, they decided they needed a high-level programming language. In 1969, NASA contracted Intermetrics⁸ to develop a new programming language, High-order Assembly Language/Shuttle or HAL/S⁹

The software was expensive; the cost was vastly underestimated. Originally, NASA thought the cost for development of the Shuttle software would be around $20M. In the end, NASA spend $200M for the original development (and that's 1970 dollars). Not surprising, over 50% of the PASS modules changed during the first 12 flights in response to requested enhancements. By 1991, the Agency had spent a total of $324 million. In 1992 NASA estimated it was spending $100M a year for maintenance of the onboard software.¹⁰

It's easy to understand why the maintenance costs were astronomical. The system was very complex and difficult to understand. Each release had to be free of any mission-ending errors. Both factors would drive testing budgets. Not only because large testing teams running extensive test protocols were necessary, but because the hardware and software required for development and test was not commercially available. Hardware had to be stockpiled, salvaged or reengineered from scratch at a significant cost. Similarly, all the software development tools had to be custom built and all programmers had to be trained at agency expense. (No university was churning out cadres of HAL/S programmers.)

Despite all that expense, the software was not bug free. According to Levison, during the first ten years, 16 'severity 1' errors were found. Of those, eight of remained in the flight code—operators simply avoided command that might trigger these errors. In addition, there 12 errors of lower severity that occurred during flight.

Not all errors occurred in the early days. In April 2009, a serious software communications error occurred in flight a few minutes after Endeavor reached orbit. So happens that bug was introduced in 1989 when a warning about code misalignment was inserted in the code. In other words, despite the Fishman's panegyric, the shuttle software was never bug free.

---

The "...Write..Stuff" is not entirely false. It has elements of truth which, to my mind, only serve to lend credibility to the otherwise misleading and potentially pernicious claims about "perfect software." It's no mere detail that the "bug free" software had been under maintenance for 25-years. The "...Write...Stuff" team was not doing development. It did not even have to deal with the major maintenance bug-a-boos of migrating to new hardware, operating systems, or compilers.

In my experience, managers who haven't programmed are unable to grasp the fundamental differences between development and maintenance tasks. They tend to be content with the idea that one size fits all. (see The Maintenance Mindset.) In other words, since the approach used by the on-board team produced "perfect software," there was and is a tendency to believe that the on-board shuttle development methods should be applied to all NASA software tasks.

I experienced fall out from this mindset while working on the Constellation Program (Cx). In the beginning of the project, we were full of hope. We were developing plans to use modern software techniques like use of software architecture and product-line concepts, an advanced computing system in the avionics package and the use of modern fault analysis techniques. These had all been used successfully in DOD or European projects. But our efforts were in vain. The project leaders had adopted a rigid philosophy like the one described in the "...Write...Stuff." All the new approaches were vetoed. It would be business-as-usual. The only software engineering team on the project was defunded.

I often wonder how things might have been different if the Cx Project Manager had absorbed the basic truth about software development that was captured so handily by John Garman in his article about the development of the shuttle software.

"If there are lessons to be learned from "the bug", they must be in how we view ourselves and our task. Building software in a large system against fixed schedules is not conducive to "bugfree" products. We can minimize the errors, and we can minimize the flight criticality of the ones that remain...but we can't treat it like a problem with a methodological solution." — Jack Garman, "The Bug Heard 'Round the World" (1981)

Despite all the tumult that the cancellation of the Cx Program created in the Agency, from a software perspective, it's a good thing it was stopped. The management did not have a realistic understanding of the software challenge they were facing. By contemporary standards, the Shuttle's PASS software would not be considered a 'big' package. Estimates at the time sized the Cx software in 100's of millions of lines of code. Using the methods described in the "...Write...Stuff," those software development efforts would not have converged—even with an infinite amount of money. An unimaginable failure was in the making.

If you are working on the development of a large software system, you can only hope that your manager doesn't read the "Write Stuff" or if she does, she knows it's irrelevant. Better she should read the account of the team that built the the HAL/S compiler. It's much better advice:

Software is a very unusual industry. You can run an assembly line with a whip, although American managers are belatedly realizing that there are better ways...Formal devices and management tricks can aid or impair it, but the impetus must come from within. The collective mind and will of the technical staff is the essence...If management strays too far from the will of the workers, tries to shape it into something that it is not, the only possible outcomes are chaos or outright rebellion. — Tony Flanders, A History of Intermetrics

But then who knows, maybe, one day, someone in senior NASA management will grasp its own lessons learned, so that, once again, NASA can do the right stuff.

---

^1. Fishman, C. "They Write the Right Stuff." Fast Company. 1996
^2. Levison. "Software and the Challenge of Flight Control" (Chapter 7), Space Shuttle Legacy: How We Did It/What We Learned AIAA. Edited by Roger Launius, James Craig, and John Krig. 2013.
^3. He tells me that out of a staff of 400 only 50 were allowed to write code.
^3a. Remember that Brooks's proposal most included non-technical staff. See "Fly in the ointment."
^4. First described in the 1968 publication "Software Engineering, Report on a conference sponsored by the NATO SCIENCE COMMITTEE."
^5. The additional effort is typically mandated the empty reassurance that "you have to do the work any." Actually we didn't.
^6. The following is based on material found in four excellent sources:

a) Tomayko's Computer in Spaceflight: The NASA Experience, Chapter 4
b) Levision's Software and the Challenge of flight Control
c)Lethbridge's Spaceline.org. http://www.spaceline.org/rocketsum/shuttle-program.html
d) Mattox and White's Space Shuttle Main Engine Controller

^7. Garman was Deputy Chief of the Spacecraft Software Division at JSC and played a key leadership role in shuttle software development. See Garman, J. "The Bug Heard 'Round the World," Sofware Engineering Notes. Vol. 6. No. 5. October 1981.
^8. Intermetrics was formed by a group from Draper Labs who had worked on the Apollo software. The story of Intermetrics is fascinating. One of the principles, Tony Flanders, has posted a very interesting history. see http://www.whysheep.com/i2/daf-history2.html
^9. According to Tomayko, 2001 a Space Odyssey was playing in the theaters at the time the new language was contracted. Perhaps Kubrick's film influenced the name.
^10. Id. Levison.

A fly in the ointment

From Chapter 3: The Surgical Team

Excerpt
...one does the cutting and the others give him every support that will enhance his effectiveness and productivity.(p. 35)

Brooks says that some jobs are too large for a small team. However, if you have a large team, Brooks' Law kicks in adding significant communication overhead, bogging down progress and bloating cost and schedule. What's the manager of a large software project to do?

Brooks' Law

Brooks has a grand plan. In subsequent chapters (4, 5, 6 and 7), he will offer a solution for managing the large project. This remedy is built on the Surgical Team concept.

What is the Surgical Team exactly? Brooks spells it out in Chapter 3 (see Picking the Archetypes for the Surgical Team for a summary).

There are a two key concepts of the Surgical Team.

• The job gets divided up into tasks that can be handled by a "10-man programming team"
• Each task produces a system is the "product of one mind," the Chief Programmer. As Brooks put it, "one does the cutting."

To make this work, Brooks prescribes the following:

• The Chief Programmer will be supported by 7 specialists (plus 2 secretaries) who are each responsible for some part of the "design and implementation"
• Each specialist communicates to the rest of the team through the Chief.¹ This cuts the number of communications path by over 80%.
• Of the 7 specialists, only one, the co-pilot, is a programmer and he "is not responsible for any part of the code."

In other words, it appears that Brooks is suggesting that for each team of 10 there be only ONE programmer. To first order this means that a staff of 100 is needed to support 10 programmers. As Brooks puts it, "So it is possible to put 200 people on a problem and face the problem of coordinating only 20 minds..." (page 37.)

Admittedly the Surgical Team as described is anachronistic. Many of the duties that previously required staff, like the 'Program Clerk,' are now done with software tools. But the main point is that the Chief is king and the co-pilot is prince; together they rule a small kingdom of specialists who do not program. A variation of the Chief-Copilot tuple is promoted by devotees of Agile Programming as pair programing.

But remember, the purpose of the surgical team is to provide an organizational remedy that solves the 'big team' problem for projects that require 10's if not 100's of programmers. (see Small is beautiful, but not applicable.) Here's where the Surgical Team salve gets sticky.

What happens in the common case that a the job cannot be partitioned into a thin layer of one-man tasks? Does the Surgical Team recurse into a hierarchy of Surgical Teams? Would you restrict a Chief Programmer on one leaf task from talking to a Chief Programmer on another leaf task? If not, the communication paths would multiply geometrically obviating the need for the surgical team.

On the other hand, if all communication was routed up the Surgical tree, there would most certainly be a communication and decision bottleneck that would be worse than any cure. As Brooks puts it, "'Schedule disaster, functional misfits, and system bugs all arise because the left hand doesn't know what the right hand is doing.' Teams drift apart in assumption" (page 235)

We had just that sort of bottleneck on my last NASA project. Our chief engineer insisted on being part of every decision. In just a month, our work ground to a dead standstill; our obligations did not. In order to the make the progress needed to meet our commitments, our experienced programmers simply completed the work without discussing it with the technical leadership. It was the only rational thing to do.

You won't find a Chief Programmer in the group

And, then there's the question of how to staff a large project. Brooks was clear that the product should only be programmed by Chief Programmers with possible small contributions from the Copilot. (So, you might say the Surgical Team has 1.1 programmers.) He describes the Chief as a developer with "great talent, ten years experience, and considerable systems and application knowledge..." (page 32). How would you staff such a team? Where would you find 10's or 100's of men and women with that skill level? And, if you did find them, how could you retain them between assignments?

It's possible that Brooks does not really mean there should only be 1.1 programmers on the surgical team. In chapter 7, Brooks discusses how senior programmers, development programmers, project programmers, junior programmers and trainees would fit into an organization. Why bother with these Indians if all you need is Chiefs? Of course if he's not serious about 1.1 programmers per team, then the bug-a-boo of Brooks Law reasserts itself.

Here's the rub. There is no easy way around Brooks' Law. The Surgical Team does not alleviate the large-team problem. There's a fly in the ointment.

I suspect when the Mythical Man-Month was written, a Surgical Team with 1.1 programmers made sense. Today, it's a flawed concept. So for purposes of this blog, I'll stretch the meaning of the Surgical Team to include as many programmers as the Chief Programmer needs to complete the task on time. Of course that means there may be no escape from Brooks Law.

---

^1. For a diagram, see the online version Text page 36 which is pdf page 50.
^2. Brooks defines architecture as "the complete and detailed specification of the user interface." (p.45) In this case "user interface" refers to a programming interface. Today, software architecture refers to a a much broader scope of concerns.