Must it be?

Continued from Riding the bow wave.

Shamelessly altered excerpt
The test effort usually exposes important gaps... Why? ... 'interface management.'

From the score of Beethoven's String Quartet No. 16 in F major (Op. 135)¹

Marking for the first chords of the last movement
of Beethoven's String Quartet #16 in F major (Op 135)

Beethoven opens the last movement of his last significant compositional effort with the question: Must it be? He concludes: It must be, but should we?

Thanks to a timely mishap on the latest International Space Station (ISS) resupply mission, I'm in an existential frame of mind.

In recent postings I've been ruminating about the ad hoc nature of testing. Thanks to an ex-colleague, who is apparently omniscient about all-things NASA, I learned about this timely existence proof of test fallibility. He sent me packing to the Orbital Sciences Corporation web page for reports on the status of the Cygnus rendezvous with the Space Station. Here's the link: Antares/Cygnus Updates.

Cygnus is a cargo carrier; Antares is the launch vehicle. The Cygnus/Antares combo is latest "commercial" solution for ISS resupply to come down the pipe--the SpaceX offerings, Dragon and Falcon, were first. The Orbital system is comparable to SpaceX, but Orbital lacks a gifted self-promoter and is less well known.

Orbital and SpaceX are the two surviving contenders in NASA's Commercial Orbital Transportation Services (COTS) Program. Both Orbital and SpaceX have received substantial funds from the Agency. Prior to 2011, Orbital received $260M and SpaceX received $376M from NASA. The combined COTS budget for FY12/13 was around a $billion (FY12=$400M, FY13=$525M), which I'm sure was judiciously split between Orbital and SpaceX.² If you add it up, that's in the vicinity of $1B apiece. (But not quite even-steven.) If this is 'commercial', what would you call 'government'? I digress.

Orbital's initial launch of Antares/Cygnus was on September 19. Cygnus was scheduled to rendezvous with the space station on September 22. Here's a few telling excerpts from the Orbital status page.

September 21, 2013
As of mid-day today, Cygnus continues to perform...remaining on track for its rendezvous...tomorrow morning.
7:45 a.m. Sunday, September 22, 2013
Following the discovery of a data format discrepancy...today's rendezvous with the station was postponed...A software update has been developed and will be tested on a ground-based simulator during the day on Sunday.
September 22, 2013
This morning, at around 1:30 a.m. EDT, Cygnus...found that some of the data received had values that it did not expect, causing Cygnus to reject the data. This mandated an interruption of the approach sequence.
Monday, September 23, 2013
...Orbital and NASA together decided to postpone the approach...Over the past 24 hours, the Orbital team developed and tested a software fix for the data format mismatch...This new schedule will allow the Orbital operations team to carefully plan and be well-rested before restarting the critical final approach.

I have no direct knowledge of the circumstances that lead to the defects in Cygnus data processing, however I do know a bit and could surmise the following:

• The discovery of the data problem was well past the 11th hour.
• The Cygnus/Antares systems under went thousands of hours of testing prior to launch. It's likely the wrong test data was used.
• The ISS ICD has been in place for decades and successful used by the both American and Russian vehicles.
• The ISS telemetry data format does not conform to the most commonly used international standard⁴
• The Orbital engineering team most surely competent and staffed with talented engineers
• The energy expended in handwringing at Orbital Ops must have enough to power an village in Virginia. Everyone is exhausted.
• The Orbital management has reported the results accurately but with a tone that conveys there has been no departure from the routine.

Why do these things happen?

Last night, I had dinner with another buddy from my NASA days. We discussed the Cygnus data problems and he pointed out that "they had problems building the pyramids. Why should things be different." He is right of course; nothing unusual here. No engineering effort will be free of problems.

But, should we simply accept the status quo of test practice as the inevitable? Is testing finally and ultimately doomed to be an ad hoc, incomplete, finger-crossing process? Should we simply shrug and accept it must be this way?

I hope not.

How can we progress to the next phase of building vastly more complex systems if there is no progress in the way we approach our objectives. Especially testing! What if we were still confined to the building techniques used by the Egyptians? There would be no arch, no concrete, no flying buttresses, no use of glass, iron and concrete. Without those innovations there would have been no Roman aqueducts, coliseum or Parthenon. No Haj Sofia. No Notre Dame. No Crystal Place. No Chrysler building. No Astrodome.

It's not that we should expect the end of problems, but that we should rue complacency. In many respects the Cygnus problem is a routine matter. But, it is the very fact that it is considered routine that causes dismay. Must it be? Let's hope it's not.


^1. If you want to hear the chords, jump to 16:45 in the recording. But, think twice! This is one of Beethoven's masterpieces.
^2. 2102 funding and pre-2012 funding comes from the NASA FY13 budget request. The 2013 budget was reported in the Space Politics Blog.
^3. These links tend to be broken.
^4. “The good thing about standards is that there are so many to choose from.” Andrew Tenenbaum