If greatness cannot be planned, why worry about hindsight?
- Sometimes, algorithms for innovation are better at finding things they’re not looking for — and this applies to the real world as well.
- Disappointment is the wrong lens for evaluating experimental exploration, because the consequent regret implies that you made a mistake.
- If new frontiers of possibility open up, then the original pursuit can be considered a worthwhile stepping stone.
Because I wrote a whole book about the benefits of following your interests without a clear objective, the opportunity to reflect on what that means for hindsight seems interesting. After all, it’s easy to regret when you fail to achieve your objective, but what exactly do you regret when you had no objective? Perhaps more deeply, if following your interests without a clear objective leads you to a disappointing place, should you feel regretful? What counts as a mistake when you have no clear objective to begin with?
The book, Why Greatness Cannot Be Planned, was co-authored with Joel Lehman and published in 2015. Its central premise is based on research into open-ended artificial intelligence (AI). We found that in some cases, algorithms for discovery or innovation are better at finding things they’re not looking for! In other words, sometimes, you can only find things by not looking for them. This insight was particularly interesting because it applies not only to algorithms but to the real world as well.
Because it’s counterintuitive and paradoxical, let me give a little extra context to clarify why and when it makes sense. In the pursuit of very difficult problems, such as curing cancer, creating artificial general intelligence, or achieving unlimited cheap and clean energy, it’s often the case that the stepping stones that lead to these kinds of outcomes are deceptive. In other words, they appear to be leading somewhere other than the objective we hope for. For example, someone researching tree sap could make an instrumental discovery for clean energy, even though researching tree sap seems entirely unrelated and therefore is unlikely to be funded by an initiative directed at clean energy. So we’d call researching tree sap in this case a deceptive stepping stone along the road to clean energy.
In the pursuit of very difficult problems, such as curing cancer, creating artificial general intelligence, or achieving unlimited cheap and clean energy, it’s often the case that the stepping stones that lead to these kinds of outcomes are deceptive.
This insight (relevant both in AI algorithms and the real world) carries significant implications for how we pursue ambitious discoveries. It shows that we absolutely need to allocate some effort towards pursuits that are simply interesting in their own right, even though we don’t yet know or can even try to predict what the payoff might be. In effect, by investing in some pursuits simply because they’re interesting, we are collecting stepping stones that may end up leading to something useful or valuable in the future. So collecting such stepping stones is its own virtue.
Why Greatness Cannot Be Planned explores this implication in significant depth. For example, it examines institutional implications, educational implications, and even personal implications (e.g., not all of your pursuits warrant a clear objective justification).
Accordingly, I have become more willing to take leaps of faith based on intuitions without a clear sense of where they lead. It’s important of course when taking such a leap to realize that it entails risk — it may not pay off. However, this view is mostly forward-looking. It advocates exploration for the sake of exploration as long as you are aware of the risks. But how do you evaluate such activities when looking back on them and their results?
In the context of hindsight, it’s interesting to ask what lessons make sense to take from a lack of payoff when you understood the risks going in and indeed embraced such risks intentionally.
Suppose you decide to build a website with novel functionality but no clear commercial prospect. People ask you what you expect to gain — will it eventually make money somehow? But you explain that you don’t know and you’re only doing it because it’s interesting to try. So you spend the next six months building the site and then one day release it.
Now imagine that what happens next is nothing. Few people visit, and even fewer care. The impact is near zero. You’ve learned only that your idea turned out to attract little attention and you’ve also gained little. The question is, how should you evaluate your decision to build that website in hindsight?
This question is surprisingly tricky. If you had set out with the objective of making money, it would be at least somewhat easier to answer because you could conclude that you failed at your objective. But in the present scenario, your only motivation was to pursue your curiosity, and that’s what you did. Can that be a failure?
If taking risks to try things because they are interesting ends up being evaluated as a mistake, then the lesson you will take is to stop following the interesting, and your world of possibilities will shrink. That cannot be the right conclusion.
While it’s true that the outcome was disappointing, I think focusing on the disappointment is the wrong lens for evaluating experimental exploration because the consequent regret implies that you made a mistake, and regret is meant to help you learn from mistakes. But if taking risks to try things because they are interesting ends up being evaluated as a mistake, then the lesson you will take is to stop following the interesting, and your world of possibilities will shrink. That cannot be the right conclusion.
After all, the fact that nothing may come of it is built into the idea of pursuing curiosity for its own sake. That is, all interesting pursuits entail risk. Because interesting ideas often imply uncertainty, if you can’t tolerate risk, then you can’t do anything interesting. So risk is intrinsic to following your interests, which means that if you made an informed decision to follow your interests, you considered the risk worth it from the start.
In that sense, if nothing useful comes of the exploration, you don’t want to end up internalizing the lesson that you should never take risks. Otherwise, you would simply stop exploring. The real mistake is the assumption that exploration is a mistake if it doesn’t lead to an objective payoff.
(Note that if you failed to recognize the risks at the outset, that would indeed be a mistake, but a different kind of mistake. It is undoubtedly important to understand what you’re risking when you initiate any new project, but as long as you do evaluate and accept the risks, then the results should not end up inhibiting you from exploring again.)
That doesn’t mean you can’t still learn to improve. You can become more informed about a subject so that you can become more sensitive to what’s interesting. For example, it’s hard to know what’s interesting in the field of AI if you don’t know anything about AI. Ultimately, interests are subjective, so you have to grapple with the idea that you can improve at a subjective pursuit. While the broader culture largely rejects subjectivity as a valid compass for decision-making, when you are deciding what’s interesting, subjective intuition or “gut instinct” is critical and deserves the chance to mature and develop.
Hindsight and regret have been on my mind recently as I contemplate the future of Maven, the serendipity-based social network I founded in 2023. I took a major personal risk to start Maven, diverting my time and energy into something different from the role of AI research scientist where I built my career. To be transparent, I started Maven hoping for some kind of success, which is an objective notion. But I also made the decision to pursue Maven in part because it was too interesting to pass up. It combines ideas from my research career with the broader cultural issues raised in my book and potentially positive social consequences. In other words, creating a serendipity-oriented network like Maven could lead in many different possible directions, so to a large extent, I’m not sure where it will lead. But I do believe that almost all of them are interesting.
So the question naturally arises, now in personal terms, if Maven eventually were to fail — for example, if we fail to secure further funding to support its growth — would it have been a mistake in hindsight to pursue it? Would I go back 18 months and tell myself not to do it?
Based on the philosophy I’ve shared here, which is a significant piece of my career and identity at this point, clearly I’d have to say no. It would not have been a mistake. I knew I was taking a risk, but I also knew it was interesting — interesting enough to be worth trying. And even if it fails, that interest will still be my truth. There are many lessons learned from such an experience, and I would be wiser for it, but one lesson I will not learn is that exploring the interesting when you know and accept the risks is a mistake in hindsight.
The key insight is that when you are pursuing something because it’s interesting, then payoffs are likely to be unpredictable.
To close, let’s finally consider when exploring the interesting actually does pay off. How can you evaluate the payoff in hindsight when there was no objective to begin with? The key insight is that when you are pursuing something because it’s interesting, then payoffs are likely to be unpredictable. In the example of the website, while the website itself may not have had any impact, you may end up with a new skillset or unique set of tools that opens up a completely unexpected opportunity.
What interestingness does is open up new playgrounds — new frontiers of possibility, even though we don’t know what we might find there. If any new playground opens up, then the original pursuit was a worthwhile stepping stone. It may not have paid off directly in its own right, but it led to even more interesting things, which is what open-ended exploration is all about.
So I’d ask myself in hindsight whether any new and unexpected playgrounds were opened up, which might even encompass a pivot in an unexpected direction along my original road. If you do find a new playground, then in a sense the exploration should be seen as a success in hindsight, because that’s really what exploration is all about. In the case of Maven, I’m confident that whatever happens with the company, it has indeed opened up new playgrounds.
