Close

Not a member yet? Register now and get started.

lock and key

Sign in to your account.

Account Login

Forgot your password?

Does Independence Matter in Market Research Data?

An interview with Professor Art Swersey of the Yale School of Management

Blueshift Research asked Professor Arthur Swersey of the Yale School of Management and his colleague Professor Johannes Ledolter of the University of Iowa to investigate whether data from independent sources would increase the likelihood of accurate forecasts in a market research project. The following is a brief summary of some of the results of that research, which is ongoing and aimed at producing an academic research piece.

Q. Professor Swersey, how are you? Last month, Blueshift Research asked you to examine theoretical justification for the idea that interviewing experts whose predictions are independent will lead to a greater likelihood of right answers than speaking with experts whose predictions are highly correlated. What did you find in looking at this?

A. I’m fine, Craig, and thanks for asking us to examine this very interesting issue. Let me start by defining correlation, which is a measure of the tendency of the outcomes of variables, such as expert predictions, to go together. An example of a situation of high correlation would be the predictions of experts in the same group, such as technology writers.

Now let’s look at two examples, one highly correlated and one not. Consider the case of two experts. If one expert’s prediction of success or failure does not depend on the other’s prediction, then the two predictions are independent. Independent predictions have correlation = 0. Now suppose there is perfect positive correlation (correlation = 1) between the predictions of the two experts; that is, if one predicts success, it is certain that the other will predict success, while if one predicts failure, the other also will predict failure. If each pair of expert predictions has perfect positive correlation, then all the experts will predict success or all will forecast failure. Therefore, the greater the correlation, the greater the tendency for experts to make the same prediction.

In examining your question, my colleague Johannes Ledolter and I found a very considerable benefit in having expert predictions that are independent or nearly independent compared with expert predictions that are highly correlated. In other words, the independent or nearly independent expert predictions will have a much greater likelihood of being on target and are a key factor in determining the success of a market research project.

Q. How did you go about trying to figure this out?

A. We used probability theory. Without going into the details of our analysis, it is pretty easy to see the advantage of independent predictions compared to predictions that have perfect positive correlation. In the perfect correlation case, knowing the prediction of any one expert tells us the predictions of all the others; they will either all predict success or all predict failure. In this case, one expert provides us with as much information as many. Each additional expert provides no further information. On the other hand, if the predictions are independent, then each additional expert will provide further useful information. As a concrete example, consider the following: A new product is being introduced. Before any interviews with experts, we estimate the product is equally likely to succeed or fail. Suppose each of our expert predictions is correct 60% of the time and that we interview six experts. In the case of perfect correlation, if all of the experts predict success, the odds of the product being successful increase but not by very much.

Specifically, we determined the likelihood that the product succeeds increases from 50% to 60%. Similarly, if all the experts predict failure, the likelihood of success decreases from 50% to 40%. (Note that the same result would hold for any number of experts.) By comparison, if most of six independent experts predict success or if most predict failure, the likelihood of success (or failure) will change by a much larger amount. For example, if all six independent experts predict success, we found that the likelihood of the product’s success increases from 50% (before the expert predictions) to more than 90%.

In reality, predictions will neither be independent nor perfectly correlated. Using our probability models, we compared the case of nearly independent predictions (correlation = 0.1) to the case of highly correlated predictions (correlation = 0.7). We found that the advantage of the low correlation case compared to the high correlation case is not as great as in the independent versus perfect correlation case, but the advantage is still extremely large.

Q. What insights do you have on how this can help investors?

A. In relation to the above example, information from nearly independent experts rather than from highly correlated experts provides much stronger evidence of likely success or failure. In the nearly independent case, if most experts predict success, the investor will be very confident the outcome will be positive, while if most experts predict failure, the investor will be very confident that the outcome will be negative. An investor’s decision will be consistent with the experts’ judgment, betting with the company when most experts predict success and against the company when most predict failure. We looked at the expected average return to an investor from nearly independent experts compared with experts whose predictions are highly correlated. We found the expected returns would be far higher in the case of experts who are nearly independent.

Thanks, Art. That is extremely beneficial and helps us make the case that a market researcher should include the variable of interdependence of experts’ predictions when looking for answers to investment questions. We look forward to seeing the full academic piece!

For more information you can contact Professor Swersey at: arthur.swersey@yale.edu.