I happened to be listening when Radio 4’s “Today Show” had a little debate about the probability of getting a pack of six double-yolk eggs. Tim Harford, who they called to help them sort it out, relates the story here.
So there are two thinking styles here. One is to solve the probability problem as posed. The other is to apply some common sense to figure out whether the probability problem makes any sense. We need both. Common sense can be misleading, but so can precise-sounding misspecifications of real world problems.
There are lessons here for the credit crunch. When the quants calculate that Goldman Sachs had seen 25 standard deviation events, several days in a row, we must conclude not that Goldman Sachs was unlucky, but that the models weren’t accurate depictions of reality.
One listener later solved the two-yolk problem. Apparently workers in egg-packing plants sort out twin-yolk eggs for themselves. If there are too many, they pack the leftovers into cartons. In other words, twin-yolk eggs cluster together. No wonder so many Today listeners have experienced bountiful cartons.
Mortgage backed securities experienced clustered losses in much the same unexpected way. If only more bankers had pondered the fable of the eggs.
The link Tim gives in the middle of my quote is to this piece, also by Tim, at the FT. Here’s the bit that Tim is referring to (emphasis at the end is mine):
What really screws up a forecast is a “structural break”, which means that some underlying parameter has changed in a way that wasn’t anticipated in the forecaster’s model.
These breaks happen with alarming frequency, but the real problem is that conventional forecasting approaches do not recognise them even after they have happened. [Snip some examples]
In all these cases, the forecasts were wrong because they had an inbuilt view of the “equilibrium” … In each case, the equilibrium changed to something new, and in each case, the forecasters wrongly predicted a return to business as usual, again and again. The lesson is that a forecasting technique that cannot deal with structural breaks is a forecasting technique that can misfire almost indefinitely.
Hendry’s ultimate goal is to forecast structural breaks. That is almost impossible: it requires a parallel model (or models) of external forces – anything from a technological breakthrough to a legislative change to a war.
Some of these structural breaks will never be predictable, although Hendry believes forecasters can and should do more to try to anticipate them.
But even if structural breaks cannot be predicted, that is no excuse for nihilism. Hendry’s methodology has already produced something worth having: the ability to spot structural breaks as they are happening. Even if Hendry cannot predict when the world will change, his computer-automated techniques can quickly spot the change after the fact.
That might sound pointless.
In fact, given that traditional economic forecasts miss structural breaks all the time, it is both difficult to achieve and useful.
Talking to Hendry, I was reminded of one of the most famous laments to be heard when the credit crisis broke in the summer. “We were seeing things that were 25-standard deviation moves, several days in a row,” said Goldman Sachs’ chief financial officer. One day should have been enough to realise that the world had changed.
That’s pretty hard-core. Imagine if under your maintained hypothesis, what just happened was a 25-standard deviation event. That’s a “holy fuck” moment. David Viniar, the GS CFO, then suggests that they occurred for several days in a row. A variety of people (for example, Brad DeLong, Felix Salmon and Chris Dillow) have pointed out that a 25-standard deviation event is so staggeringly unlikely that the universe isn’t old enough for us to seriously believe that one has ever occurred. It is therefore absurd to propose that even a single such event occurred. The idea that several of them happened in the space of a few days is beyond imagining.
Which is why Tim Harford pointed out that even after the first day where, according to their models, it appeared as though a 25-standard deviation event had just occurred, it should have been obvious to anyone with the slightest understanding of probability and statistics that they were staring at a structural break.
In particular, as we now know, asset returns have thicker tails than previously thought and, possibly more importantly, the correlation of asset returns varies with the magnitude of that return. For exceptionally bad outcomes, asset returns are significantly correlated.