Over in the Nov. 2011 Scientific American, Wrong author David H. Freedman has an article titled Why Economic Models Are Always Wrong, in which Freedman builds off a 2005 paper (pdf) by earth scientist Jonathan N. Carter et al. from a conference on Sensitivity Analysis of Model Output; the paper's titled "Our Calibrated Model has No Predictive Value: An Example from the Petroleum Industry" (Freedman, pers. comm.). The SciAm article starts with this geophysical model, draws conclusions about economic models, and mentions climate models not at all.
If I understand correctly, Carter and colleagues found that when they tweaked parameters to "hindcast", i.e. to fit the existing dataset perfectly, that multiple sets of parameter values would equally well fit these existing data - yet, having different parameter values, they'd diverge in their future forecasts, and thus, Freedman says, economic models are inherently flawed.
Some commenters on the article are taking this "econ models are inherently flawed" argument to mean that, because climate models are calibrated(?validated?) via hindcasting, they too are dubious.
I asked climate modeling software prof. Steve Easterbrook to weigh in on this, & his essential answer was basically no because climate models are physics-based, unlike economic models; but yes, the same hazard does lie in wait and must be avoided:
"The phenomenon described is completely correct. If you use empirical methods to tune the models, you’re in danger of getting bad forecasts. For models of the physical climate, this is avoided by paying more attention to the underlying physics – instead of empirical tuning, you work on understanding the underlying processes, and improving how they’re captured in the model. Then you do lots of different hindcasts, process studies, etc, to check how well you did. One of the problems with economics models is there is no set of basic physical principles to underpin them. Which makes it much harder to avoid the problem."Commenter SteveO (#23 &24) concurs, concluding:
"this is why climate models are pretty good predictors of reality (and global warming). They are not just randomly fit numerical models, they are based on physical principles, thus usefully constraining the formula-space to ones that adequately model reality."And commenter Patricio Parada (#36) points out that you use two separate sets of data, the first to "train" the model, the second to assess how good the model is:
"one should divide the data in two subsets: a training set for model computation, and a validation set, for generalization study, and compute the fitness of the model to both sets. Only then, I have all the ingredients needed to select a model that explains both the training data and generalizes well."And Dr. Carter says:
(tbd)





0 comments:
Post a Comment