Small is beautiful, but not applicable

From Chapter 3: The Surgical Team

Excerpt
...one continually hears young programming managers assert that they favor a small, sharp team of first-class people, rather than a project with hundreds of programmers... But this naïve statement of the alternatives avoids the hard problem...the small, sharp team concept...is too slow for really big systems.(p. 30-31.)

From Small is Beautiful¹

Excerpt
One of the most fateful errors of our age is the belief that "the problem of production" has been solved. — Opening sentence of "Problem of Production (Chapter 1)

In chapter 3, The Surgical Team, Brooks takes on the topic of how to organization and manage a very large team. It's a hard problem. A very hard problem.

Why bother? Why not simply deploy a small team composed of the best people? Brooks cites a study from the 60's where highly skilled programmers were found to be 10X more productive that average programmers. More recent estimates put that ratio at around 4X.² In either case, the numbers are compelling. As if that wasn't reason enough, there's Books' Law : as the team gets larger, the number of communication paths grow. And, of course, the added complications that stem from decomposing the system into parts that can be developed on parallel paths. (See Humpty-Dumpty Effect).

What's not to like? Why would anyone, ever, approach a project with a large team?

IBM System/360

 Brooks offers this compelling analysis based his experience on the IBM System/360 Project³.  Brooks estimates that, if you included all members of the development organization⁴, the System/360 took about 5,000 man years to build. He recalls that the original project plans called for only 200 people. However, at one point, there was over 1,000 people working on the project. He then surmises that "if "men and months traded evenly" the project would have taken 25 years. In other words the System/360 wouldn't have delivered until the late 80's when PCs and workstations dominated the marketplace; in which case, the System/360 would have been embarrassingly obsolete and a certain business disaster.   

What if the system was built by a small, highly talented team? Would that have been preferable?

Brooks offers the following analysis. He assumes a team of 10 since that is the largest practical size for a small team.⁵ For the sake of discussion, he assumes the staff on the small team is 7-times more productive than people who actually did the System/360 job. And, in order to make the most favorable case for a small team, he throws in a 7X bonus because there are fewer communication channels. In other words, he's going to assert that the small team can produce products nearly 50-times as fast as a large team. The result: the small team gets the job done in just 10 years. However, that's over twice as long as the actual project. In other words, the System/360 would have come to market just about the time DEC introduced the VAX which was a significant advance over the IBM 360. Once again the result would have been a business disaster for IBM.

Note that Brooks' analysis does not account for the new technologies that came to market during this period.⁶ If you bear those in mind it's likely the System/360 project would have been stuck in a cycle of endless catch-up and never been completed. In other words, a large team was necessary for the System/360 to be a viable product.

---

The development of a space system also requires a large team. Development of the integrated flight/ground systems is a large-scale undertaking requiring coordination between many disciplines. For the sake of discussion, just consider the software; it will provide some insight into the scale of the undertaking.

Here's a partial list of software teams who typically participate in the engineering of a mission.

Collage of NASA Teams

• A flight software team that typically includes managers, software-system engineers, programmers, and testers.
• A flight hardware team that uses programming languages (i.e. software) to build the computer components used on spacecraft⁷. The hardware team typically includes managers and electrical engineers.
• Several teams of engineering discipline analysts. The analysts participate in both the development and operation of the spacecraft. These displines include attitude control, power, navigation, and mission planning to name a few. Some of the discipline teams are very large (e.g. navigation), some are small (e.g. power).
• Several teams of scientists who build and operate the spacecraft instruments.
• A team responsible for spacecraft command tools. This team usually includes managers, software-system engineers, programmers, and testers.
• A team responsible for the ground system integration and infrastructure. This team usually includes managers, software-system engineers, programmers, and testers.
• The teams responsible for the space communications infrastructure. There are many aspects to the communication system and, in its own right, is the size of a mission
• A team responsible for payload and booster integration and the infrastructure needed to support it. There are many aspects to the communication system and, in its own right, is the size of a mission.
• A team responsible for turning raw data in to engineering and science data products. (e.g. producing the pictures taken by the rovers.) This team usually includes managers, software-system engineers, programmers, and testers.
• The public relations teams needed to set up and maintain web sites and other social media.
• Institutional IT infrastructure teams who are responsible for setting up all the computers and networking

---

In Chapter 3, Brooks describes an approach for organizing a large teams that would make them practical. In a nutshell, he defines a hierarchical organization that reduces the communication paths. This will be the subject of subsequent postings. But, in my experience, this hierarchical arrangement does not work. It inevitably produces a bottle neck around the leadership. It often proves to a recipe for indecision since the line of responsibility is unambiguous. What's more, the guys in the trenches are often a LOT smarter than the appointed technical leads who are, too often, selected for political skills rather than technical talent.

You need a large team to build and fly a space system. We haven't made much progress since Brooks wrestled with the System/360. Large teams offer substantial challenges, many of which have no known solution and would be a suitable research topic. There is no authoritative list of capabilities that are needed to manage a large team, but here's few suggestions:

• A highly-expressive means for describing system design and system function that permits consistent interpretation at all levels of abstraction and realization. This would dramatically decrease the number of high-bandwidth communication paths.
• A robust and expressive interface description method for specifying requirements, design, signatures, protocols and the functional semantics behind the interface. This would provide a basis for a robust emulation strategy, system-wide continuous integration, and the development of early and stable test routines.
• Planning tools for efficiently developing and modifying detailed deliveries. This will provide overall visibility of how changes in one team may propagate across the development effort. ⁸
• Configuration management tools that with 'smart' support for parallel development of multiple code branches (or streams)—i.e. no hand merging. This will enable a means to project-wide continuous integration without drawing programmers away from new code development.
• Automatic and non-invasive metric collection tools that provide visibility in to progress and quality. This will substantially decrease the effort required to support management oversight.
• A novel risk assessment methodology based on an automatic and achievable testing protocol. This will provide an substantial method for understanding of the state of product without relying on the unrealistic hope of comprehensive testing

You might note that none of these suggested capabilities add functionality to the system, only to the system that's needed to build the system.

While there is a plethora of tools (e.g. IM, issue trackers, stream-based CM systems, etc.) that help facilitate current practice. As far as I know, no one has the tools or techniques needed to address the problems confronting a large development organization when it builds a large and complex system like a space system.

Some day engineering practice may evolve so that building things of inordinate complexity will be within the grasp of a small team. Meanwhile small is not practical.

---

^1. Schumacher, E.F., "Small is beautiful: economics as if people mattered". Harper & Row, 1973.
^2. In Barry Boehm's essential cost-estimation text, "Software Cost Estimation with COCOMO II," he estimates that a highly skilled team, on an "largely unprecendented project, like building a new spacecraft is slightly over 4X more productive. See Boehm, B., et al. "Software Cost Estimation with COCOMO II." Prentice Hall. 2000. p.32.
^3. Brooks was manager of the System/360 project from 1961- 1965.
^4. Books includes programmers, writers, machine operators, clerks, secretaries, managers, support groups among those he counts as part of the development organization.
^5. Brooks doesn't make it clear who is included in the 10. I assume he means 10 programmers. That would probably come to a full team of 20 as follows: 1 manager, 1 administrator, 2 Systems Engineer, 10 programmers and 6 I&T (i.e. Testers). By contemporary standards, that is NOT a small team. Here's how I would define a typical small team: 1 manager, .5 administrator, 1.5 Systems Engineer, 5 programmers and 4 I&T. In round numbers, I'd expect the annual budget (in a contemporary aerospace company) for the former to be around $6-7M. I'd expect that latter to be $3-4M.
^6. A few of the improvements include important advances in instruction sets, memory access, graphics, networks, operating systems, and programming languages.
^7. Hardware is "written" using verilog or VHDL to build devices like FPGAs . Hardware teams typically do not use conventional software engineering techniques that are considered best practice. By and large most hardware managers do not think of software development as part of the hardware development process.
^8. The need for reliable delivery of specific functionality calls into question the value of using agile development techniques in a well managed software effort. Since, by definition, the results of an agile task is unpredictable.