Is it done yet?

From Chapter 2: The Mythical Man-Month

Excerpt
...our estimating techniques fallaciously confuse effort with progress hiding the assumption that men and months are interchangeable. (page 14)

It's official now. The fallacy of confusing effort with progress is has been deemed a good management practice. It's called "Earned Value Management" or EVM. EVM is now required for DoD and NASA projects. In fact, both DoD and NASA have entire divisions devoted to the practice and enforcement of EVM.

In a nutshell, EVM is a schedule and cost tracking technique for measuring progress as planned scope and planned cost against actual progress and actual cost. The overall scope of a project is called its planned value (PV). The PV is the sum of the PVs for the individual tasks that compose the project. Progress towards completion is called earned value (EV). EV is acquired according to management-defined earning rules. If a task is reported as 50% complete, it acquires half it's EV. If that task represents 25% of the entire project, then the project acquires 12.5% EV. In other words, the project is 1/8th done. When the project EV equals 100% of the PV, the project is, by definition, done.

The EVM notion of progress is based on the tacit assumption that there is a fixed relationship between completed scope and budget.¹ Consider what happens if there's a scope change. If there is more scope, have you suddenly earned less? If there's less scope, have you already earned more?

In essence,the EVM measure of progress begs the essential question: What does it mean to be done?

So happens that determining when a software product is done is a really hard problem. Is it done when the requirements are met? What if the requirements change? Is it done when a version is released? What if the product is missing key feature or has key defects? Is it done when the product goes into maintenance? How do you decide what's a fix and what's a feature? Is it done when the product is no longer in maintenance?² How would you differentiate between done and dead? Or is it just done when the money runs out?

Good definitions of scope are usually expressed as requirements ³. During the planning phase of a task, a good software team will define the system requirements and then select a subset that fits schedule and budget as the definition of done.

Sound like a good basis for EVM? Not in my view. Here' a few reasons why:

• Requirements change. Why? You get smarter as the system is built. Ambiguities need to be clarified. Design considerations drive compromise. Estimates are wrong. Priorities change. Customer needs change. Budgets change. Schedules change. Surprises are a fact of life.
• Estimation of development effort based on requirements is very difficult to get right. (And, not only for the reasons described in the previous post called In the beginning....) Gauging the effort takes a strong intuition, deep technical knowledge and lots of practice.
• Not all development methods are equal. 'The last 10% takes 90% of the time,' was the by-word when the waterfall development model was the standard. The newer iterative/incremental models, like Agile, provide a much better indication of project progress because the systems is continually integrated and tested. Surprises come early; you know where you are in the schedule and can adopt accordingly. Unfortunately, that means the initial PV and the recorded EV are meaningless.
• Earning rules that allow EV for partial progress invite schedule surprise. In my view, software is not done until it has been tested successfully--until that point, you don't really know how close a task is to completion. It's 100% or nothing. However, in an EVM managed environment, if you don't provide an estimate of partial progress, you won't show any EVM progress until you are done. That doesn't look good to the bean counters. Consequently, almost everyone reports partial progress leading to highly unreliable estimates of actual progress and last minute schedule slips.
• EVM inhibits needed schedule changes in response to the actual state of development. Scope change invalidates PV and previously reported EV. It also triggers EVM rework by the administrative team and unwelcome review by a harried project management who is weary of the surprises coming from the software development teams. Consequently, an experience software development manager may maintain one schedule for EVM reporting and one schedule for tracking the real work. Evil lurks.

From what I have observed, EVM is not just an innocuous drain of resources. A manager whose EV falls below the curve may lose funding. In a famous incident, the Navy's A-12 Avenger program was cancelled because of EVM data showed the project was in trouble. Even worse are the mendacious survival behaviors that have and will evolve as a EVM workarounds.

I am not advocating that software development be done without accountability, but EVM institutionalizes a technique that "fallaciously confuse(s) effort with progress." Ironically, progress can be accurately and intuitively understood by simply using iterative/incremental development methods that methods don't fit well with EVM.

EVM is just one the many new institutionalize obligations that have taken hold in the agency during the last decade. In an effort to develop policies that ensure efficiency and rigor, NASA has added cost and spawned management techniques that operate in the shadows.

¹ Without out this assumption EVM would only report how much budget was left or how much time was left on the schedule.

² Ironically, at that point the product has very little value because no sensible user would invest in an orphaned software product unless they were planning to continue development and maintenance on there own nickel.

³ Where requirements could be expressed as 'shall' statements, use cases, test cases, scenarios, story points or any form that describes what the software will do when its done.