Wisdom or Madness: Expert Data on Wisdom of Crowds

From structured expert judgment data with realizations it is concluded that (1) experts' Mean Absolute Percentage Errors are very fat tailed, making convergence problematic, (2) probabilistic proximity of experts' median forecasts to realizations are modestly dependent, whereas experts' abilities to catch realizations in their 90% bands are much less so, (3) expert agreement does not predict expert panel performance, (4) regarding the performance metrics Statistical Accuracy and Mean Absolute Percentage Errors, number of experts is helpful for the first, harmful for the second whereas dependence in placement of medians is harmful for the first, helpful for the second, and (5) following Jensen's inequality, averaging experts' median assessments is slightly better than choosing a random expert but (from a previous publication) much worse than the median of equally weighted or performance weighted combinations of experts' distributions, underscoring the importance of method of aggregation. Probabilistic crowds are wiser than point forecast crowds.

Keywords: Expert judgment, MAPE, Fat tails, Agreement, Dependence, Diversity, Explained variance.