Capturing the magic: Human Experts and AI Predict Investment Success
Ask a successful startup seed investor, “How do you do it?” and you will often hear, “You’ve got to have the magic.” Press harder and you will likely hear, “It takes years of apprenticeship and experience in pattern recognition.” The myth of expert magic was challenged decades ago but it still persists. Research in 1970, led by Lewis Goldberg at UCLA showed that:
“If you wanted to know whether you had cancer or not, you were better off using the algorithm that the researchers had created than you were asking the radiologist to study the X-ray. The simple algorithm had outperformed not merely the group of doctors; it had outperformed even the single best doctor.”
Years of experience can be captured in heuristics, rules of thumb that guide decision making and learning. By learning the heuristics used by experts and applying them in an algorithm, a systematic approach to capturing human judgment was more reliable than the experts. So how do we capture the investor magic?
Expertise when analyzed can be represented as a combination of facts and heuristics. When applied to early stage investing, one frequently encounters heuristics like “bet on the team” or “bet on the market opportunity.”
Research on early stage investing in this way done by Wiltbank and others suggests that we can establish a framework based on a distillation of best practices that correlate to the best investment outcomes. While best practice research does include analyzing market and team, there is more.
Based on prior research, one rule dominates: the more diligence the better. But diligence on what? Interviews with investors yield four key questions:
- Is there a solid, compelling business/market opportunity?
- Is this the right team to take advantage of the opportunity?
- Is there a strong network effect? Are the early investors and advisors connected and able to propel the company through their relationships?
- How strong is the conviction that this is a great investment opportunity?
We can capture this in a high-level heuristic:
Startups with strong teams in a compelling market opportunity supported by a committed and connected group of investors and advisors are winners.
Easy to describe the criteria, but much harder to assess in markets with limited information availability. In these markets, clever algorithms and data analysis are insufficient in isolation. Expert intuition, earned over years of investing, must be combined with data analysis to accurately assess the intrinsic value of a startup. CrowdSmart’s key innovation is an AI that augments, rather than replaces, the hard-won experience of experts. Here’s how it works.
The CrowdSmart process engages a diverse team of experts in an interview and collaborative process based on diligence materials and interactions with the startup team that collects the factual and heuristic knowledge about the startup in the above mentioned four categories. We determine how each startup scores on business opportunity, team, network effects and investment attractiveness. Most importantly, we dive into the why behind feedback. We ask you to numerically score the team, then provide the thought process behind the score. Further, our proprietary technology learns the most relevant issues or heuristics applied by the assessment team to correctly weight their influence.
The result of our proven knowledge acquisition process is a collective knowledge imprint, customized to each startup. Over the past four years, this process has shown to have greater than 70% accuracy in predicting funding momentum for early stage companies, a primary leading indicator for a startup’s ability to achieve a positive exit.
Each evaluation yields on the order of a few hundred quantitative data points and approximately 10,000 words per startup per assessment team. We have the following macro data based on analysis of the assessment teams’ comments and discussion:
FIGURE 1: Theme Classification — All Startups
The numbers are percentages based on frequency of comments classified in the specific category by our theme classification system. For example, 25% of the comments made in all startups evaluated so far are about the team or management, 18% about the market or business opportunity, etc. So broadly, evaluators focused more attention in their discussions on team, market opportunity, early investors/network, and product in that order. There is a long tail of other categories that make up an additional 28% of the responses divided over an additional 25 theme topics. This chart simply says that, over all assessments on the CrowdSmart platform, the number 1 topic was Team followed by Market, etc. This is not surprising since the interview process frames focus on the top three topics. Note that these frames of discussion accounted for 55% of the comments. Frequency of mention does not however equal importance to the investor. This is where CrowdSmart’s adaptive learning technology makes a difference.
If we look at specific startups which we have anonymized as Startup 1 and Startup 2, we see some variance from broad averages.
FIGURE 2: Startup 1 Theme Frequency
Note based on frequency, this company is more heavily weighted to team. If we then look at this company’s themes ranked by relevance, we see the following:
FIGURE 3: Startup 1 Theme Relevancy Percentage
The topic of the team is still on top with the highest relevance score thus agreeing with frequency of mention in this case. However, note that competition, sales and finance are highly relevant to investors in terms of priority even though they are not high in frequency of mention.
If we look at a second company, we see that product leads based on frequency of comments.
FIGURE 4: Startup 2 Theme Frequency
However, if we look at relevance for this company we find different themes rise to the top:
FIGURE 5: Startup 2 Theme Relevancy Percentage
Note that Go to Market and Potential Exits from a frequency of mention are less than 5% but both are highly relevant in driving the decision (to invest in this case). The next blog will include more context on how we arrive at the relevancy of a theme.
In summary, we have shown that by starting with an approach based on best practice research, we have validated that the heuristics identified result in a highly accurate way to assess and predict startup success. The magic can be captured. The magic can be scaled and deployed beyond tightly held investor communities. What makes the CrowdSmart process unique is:
- Technology that marries learning priorities from collaborative conversations
- In-depth natural language processing that translates qualitative insight
- Data that can be used in mathematically-based machine learning models
Predictions from collaborative conversations are tracked against real world results. This is a platform that will continuously learn and apply the “magic”.
