As the Advanced Research Projects Agency (ARPA) approach to research and development (R&D) grows in popularity, it’s a good time to take stock of the model. This column will serve as the first in a series that explores what makes the ARPA model unique and transformative – as well as what we can learn from them and apply more broadly.
Inspired by the success of the U.S. Defense Advanced Research Projects Agency (DARPA) – whose breakthroughs include the internet, GPS, stealth technologies, and mRNA vaccines – new ARPAs in sectors like health and energy have emerged. ARPAs possess a few elements that set them apart from other R&D outfits, but most notably, they conduct “high-risk, high–reward” R&D. ARPAs are risk-takers and therefore comfortable with failure. They acknowledge that failures are necessary for learning, and they strive for bold new solutions to some of our nation’s toughest challenges.
Some months ago, our team created an online forum for ARPA experts. (Let us know if you want to join!) Recently, Ben Reinhardt, CEO of Speculative Technologies, asked, “What are your ‘favorite’ ways that ARPA programs can fail?” This simple question led to a rich discussion about failure – when it’s worthwhile vs. when it’s wasteful. Here are some of the key insights.
Insight #1: An ARPA program is a failure if it could have been executed successfully by a federal agency doing more traditional R&D.
At their core, ARPAs are driven to tackle “moonshot” goals that are ahead of our time, and difficult to achieve with the structure, capacity, and design of existing federal programs. For example, DARPA was created with the mission to “make pivotal investments in breakthrough technologies for national security.” This was a bold new endeavor for the federal government.
ARPAs are uniquely designed to catalyze cutting-edge innovations. They exercise organizational independence and flexibility, prioritize interdisciplinary collaboration, and invest in brilliant program managers who are given significant autonomy. If an ARPA program could have been carried out by a federal agency that is not designed for high-risk, high-reward R&D, then it’s not a good use of ARPA resources and therefore could be considered a failure.
Christian Macedonia, CEO of iReprogram and a former DARPA program manager, affirmed this point of view. All DARPA programs achieve some results, he said, so from a scientific standpoint, it is safe to say that almost none are “failures.” However, if the funding for DARPA was given to the National Science Foundation or the National Institute of Health with the same results, then the DARPA program “failed.”
Insight #2: ARPA programs that do not achieve their “moonshot” goal can still yield innovative results and are therefore not true failures.
ARPA programs don’t always meet their “moonshot” goals. Does this equate with failure?
Although ARPAs aim to generate game-changing innovations, their impact is rarely immediate. Many ARPAs that did not meet their moonshot technical goals have produced outcomes that catalyze future innovations and greatly benefit the broader field.
One ARPA expert offered an example in the field of Defense – Have Blue, a proof of concept demonstrator for a stealth fighter manufactured by the Lockheed Corporation. Although the two vehicles were lost during the demonstration, Have Blue eventually led to the adoption of stealth technology by the U.S. Air Force.
The expert noted, “The beauty of this is that sometimes you might not reach the moonshot, but along the way, you unlock a critical insight that is paradigm shifting.”
Paul Cohen, a Computer Science professor at the University of Pittsburgh, also points to the “gulf” that exists between researchers’ aspirations and accomplishments in the field of artificial intelligence (AI). Nonetheless, Cohen said that “countless” AI projects funded by DARPA have yielded outcomes that “didn’t solve the problem” but advanced the field in positive directions.
Insight #3: ARPA failures can result from repeating, rather than learning from, mistakes.
Mistakes are a natural and productive part of the learning process, and this is no different for ARPAs. Most agree that a straightforward type of program failure lies in repeating the same mistakes made before, which signifies a lack of learning and an ineffective investment of time and resources. However, many suggest that if ARPA programs are “making new mistakes,” they should not be classified as failures.
“The only real failure is failure we learned nothing from,” said Adam Russell, former acting deputy director of ARPA-H.
Referring to the concept of “intelligible failures,” Russell argues that making new mistakes entails the process of using failures to “find a path forward.” For ARPAs in particular, this means experimenting with new ideas, taking risks in testing out predictions and hypotheses, expanding on one’s knowledge base, and eventually, sharing the learning widely to benefit future research.
Insight #4: ARPA programs must “fail well” to turn short-term failures into long-term successes.
Oftentimes, programs repeat the same paths or make the same mistakes because they were not able to learn from previous programs. This often happens due to a lack of available information and learning about how programs have failed and what challenges they encountered.
That’s why ARPAs should be incentivized to “fail well” by uncovering useful information about the projects that have hit technical dead ends and informing future projects with lessons learned.
Steven Buchsbaum, a former DARPA program manager and key leader in launching the Homeland Security Advanced Research Projects Agency (HSARPA), affirmed this approach. He elaborated on a weakness in the current innovation funding system – namely a lack of incentive and forum to “fail comprehensively” by generating and publishing data that conclusively demonstrates why and how the projects have failed. This weakness, he contends, often leads to a repeated cycle of investing in research projects unknowingly repeating paths that have been tried before.
Apart from collecting and sharing data on failures, it is also important to select or train program leadership to take the risks that will yield either meaningful success or productive failures. For ARPAs in particular, effective leadership often stems from taking the needed risks, incentivizing collaboration within and across teams, and creating a culture of trust.
While these insights apply to the ARPA model, most contain kernels of wisdom for how all of us can think about success vs. failure. Even if you’re not an ARPA program manager, it’s worth remembering that failure is not to be avoided at all costs. Failure is an opportunity to learn and do better the next time. It’s the only way anything truly innovative came to be.
– This article was written by Ulrich Boser and Yueying Yu.
2 thoughts on “Lessons on ARPA “Failures””
Just because something ls labeled “ARPA-x” for some value of “x” does NOT make it an analogue of the DARPA original. There are properties of DARPA that contribute to its successes: Program managers deeply engaged in the technology, no in-house laboratory mouths to feed, limited term Program Manager assignments, generally ample funding, tolerance for failure, potential (DOD) adopters forming a possible business model, ….
v
Vint, Thanks for the note. A huge of fan of your work. And thanks for flagging. That’s an important point. We’ve been learning from current and from ARPA PMs that the qualities you noted, as well as attributes like organizational flexibility and independence; active project management; and interdisciplinary teamwork are critical to fostering breakthrough innovations. We hope to shed more light on this topic with additional posts.