Quote of the Week, June 12, 2007
This is the second a series of quotes by
Xiao Li Meng , from an introduction to Markov Chain Monte Carlo (MCMC), given to a room full of astronomers, as part of the April 25, 2006 joint meeting of Harvard’s “Stat 310″ and the California-Harvard Astrostatistics Collaboration. This one has a long summary as the lead-in, but hang in there!
Summary first (from earlier in Xiao Li Meng’s presentation):
Let us tackle a harder problem, with the Metropolis Hastings Algorithm.
An example: a tougher distribution, not Normal in [at least one of the dimensions], and multi-modal… FIRST I propose a draw, from an approximate distribution. THEN I compare it to true distribution, using the ratio of proposal to target distribution. The next draw: tells whether to accept the new draw or stay with the old draw.Our intuition:
1/ For original Metropolis algorithm, it looks “geometric” (In the example, we are sampling “x,z”; if the point falls under our xz curve, accept it.)2/ The speed of algorithm depends on how close you are with the approximation. There is a trade-off with “stickiness”.
Practical questions:
How large should say, N be? This is NOT AN EASY PROBLEM! The KEY difficulty: multiple modes in unknown area. We want to know all (major) modes first, as well as estimates of the surrounding areas… [To handle this,] don’t run a single chain; run multiple chains.
Look at between-chain variance; and within-chain variance. BUT there is no “foolproof” here… The starting point should be as broad as possible. Go somewhere crazy. Then combine, either simply as these are independent; or [in a more complicated way as in Meng and Gellman].
And here’s the Actual Quote of the Week:
[Astrophysicist] Aneta Siemiginowska: How do you make these proposals?
[Statistician] Xiao Li Meng: Call a professional statistician like me.
But seriously – it can be hard. But really you don’t need something perfect. You just need something decent.
aconnors:
This quote is actually a comment on VLK’s May 25th post, “On the Unreliability of Fitting”.
06-13-2007, 5:58 pmIt seems to me that Xiao Li is telling us that MCMC methods are not a panacea for “fitting” (i.e. finding the modes and mapping out uncertainties) in multi-modal spaces. Considerable care must still be taken to make sure one covers the difficult parameter/probability space.
vlk:
Yes, agree completely, and in fact it reinforces the point I was trying to make, which is that blind assembly-line fitting runs on large numbers of datasets carried out without human supervision are all highly suspect.
Even something as simple as BEHR runs into trouble finding the right confidence interval for the fractional hardness ratio HR — it has a W-shaped posterior when the counts are low and the prior is aggressively non-informative, and tends to catch the edges of the range than the central bump.
06-13-2007, 9:10 pm