The Heilmeier Catechism

The Heilmeier Catechism is a tool you should use any time you start a data project.

The Problem

Scoping a data project can be hard. There are often so many “unknown unknowns” that you may not know if your project goal is even possible. This uncertainty easily leads to yak shaving and other forms of scope creep.

Every project imposes opportunity costs. There is a limited number of data scientists. GPU compute is expensive. There are only 13 weeks to meet quarterly goals. Each project and every scope change represents other projects that will not get done.

You need a tool to control scope and judge opportunity costs for your data projects.

The Tool

George H. Heilmeier, a former DARPA director (1975-1977), crafted a set of questions known as the “Heilmeier Catechism” to help Agency officials think through and evaluate proposed research programs.¹

1. What are you trying to do? Articulate your objectives using absolutely no jargon.
2. How is it done today, and what are the limits of current practice?
3. What is new in your approach and why do you think it will be successful?
4. Who cares? If you are successful, what difference will it make?
5. What are the risks?
6. How much will it cost?
7. How long will it take?
8. What are the mid-term and final “exams” to check for success?

Every proposed data project should answer these questions. They are not the only questions any project needs, but they are the questions common to every project. The questions define a prospective project in a way that it can be compared to other prospective projects to make a proper accounting of opportunity costs.

The Breakdown

These eight questions break down into 5 groups.

Objectives without jargon

Heilmeier’s first question accomplishes three things:

1. It forces the researcher to define the scope of the project. The simple requirement to state objectives naturally limits the scope of a project. Defining what a project is also tells you what it is not. Simple reference to this question and its answer can reveal whether the project is suffering creep.
2. It prevents the researcher from confusing bafflegab with an actual idea. The requirement to eschew jargon is vital. It pushes the researcher to think about issues he would normally abstract away with jargon. The researcher must consider the concrete steps to reach an abstract goal.
3. It provides a basis for the less-expert evaluator to consider the proposal in the wider context. A researcher is going to be an expert in the project. The proposal evaluator is going to be an expert in the wider context, but cannot be expert in every field. To accomplish the task of balancing opportunity cost, the researcher must communicate for non-experts to understand.

Consider previous work

Questions two and three force the researcher to distinguish the proposed project from prior work.

The second question of the catechism forces the researcher to acknowledge successful prior work. Many ideas facing this analysis step reveal themselves to be re-invention of the wheel. Even ideas that are different may not appear so to the researcher’s audience. This question aligns every party’s understanding of the current practice.

Heilmeier’s third question is similar, but directed toward prior unsuccessful work. Producing results different from current practice is not enough. The new project must be executed in a different manner from any prior failures.

Cui bono?

Where the first question is about what is being proposed, the fourth question of the Heilmeier Catechism is where the research sells the benefits of the proposed project. The answer should express who will benefit, how much, and with what impact. If the project is successful, this will be the researcher’s resume bullet point. The job of the proposal evaluator is to decide where to spend resources on bets. The researcher must show that the expenditure could have a worthwhile return.

Control costs

Questions five, six, and seven are where the researcher gives the price to the evaluator.

There is little depth to the sixth question. “How much will it cost?” is straightforward and obviously important. The main task for the researcher is to put the work in to build a proper estimate of the personnel, materials, equipment, travel, etc. needed to work the plan.

The seventh question, regarding time of execution, is likewise important. A solution arriving too late is no solution and time spent on one task means work not being done on other tasks. Some longer term projects may have lower monetary costs, but provide a greater span for risks to actualize.

Whereas questions six and seven concern the knowable costs, catechism question five forces the researcher to consider what could go wrong. Risks are potential circumstances that could increase costs to complete the project. The types of costs covered by risk analysis go beyond time and money to include reputational harm to the firm or the loss of irreplaceable resources. Forcing the researcher to consider risks has shut down many bad ideas.

Accounting for the unknown

Heilmeier’s eighth question is the most easily overlooked, but the most critical to address the “unknown unknowns” and to combat scope creep. The project proposal must include objective standards to evaluate whether the project is on track. It is necessary that there be multiple of these points of evaluation during the project and not only a pass/fail at the end. Further, it is necessary that the dates for these evaluations are fixed before the project kick-off.

In conditions of uncertainty, the researcher and the project sponsor need objective, agreed-upon criteria to decide whether a project is succeeding. Projects that are doomed to fail need to “fail fast”, so that the project sponsor can avoid throwing good money after bad. Setting these criteria in the project proposal gives everyone a fair chance to know whether a project has failed.

Researchers are often convinced they are just one small step away from a breakthrough or are convinced they can make just one more small pivot to deliver a positive result. Intermediate and final evaluations constrain these tendencies by constantly giving the researcher deadlines to meet. Fixed evaluations with deadlines remove opportunities for yak-shaving and scope creep.

The Conclusion

Choosing where to spend resources on data projects can be hard because of uncertainties and opportunity costs. The Heilmeier Catechism is a set of eight questions that every project should be able answer. It is a great tool for managing these costs and uncertainties for the full lifecycle of a project.