"Prediction is very difficult, especially if it's about the future"---Bohr

(Photo from salon.com)

Sports and political prognosticator par excellence Nate Silver, on why making sports predictions is much easier than making predictions in other areas, such as economics or politics:

Sports has awesome data: "I mean data that’s accurate, precise and subjected to rigorous quality control." Also, it's extensive; in baseball's case the statistics go back over a hundred years.

"When the recession hit in December 2007 — the worst economic collapse since the Great Depression — most economists didn’t believe we were in one at all."

In sports "rules are explicit and...we know a lot about causality." Nate Silver contrasts sports with the real-world example of earthquake prediction, where the causes are not well-known. In trying to determine correlations "there are a billion possible relationships in geology’s historical data, [and] you’ll come up with a thousand million-to-one coincidences on the basis of chance alone."

Sports offers fast feedback and clear marks of success. If sports tactics are working, results show up nearly instantaneously. In contrast one most wait four years to test Presidential election strategies.

Big data is the management fad du jour because it has yielded impressive results in non-sports areas. But non-technicians often underestimate the work necessary to screen, organize, and analyze the information.

Prediction: when big data doesn't produce miraculous insights--as it often won't because not everyone is as skilled as Nate Silver--expect disappointment.