Author Archive

Poisson vs Gaussian

We astronomers are rather fond of approximating our counting statistics with Gaussian error distributions, and a lot of ink has been spilled justifying and/or denigrating this habit. But just how bad is the approximation anyway?

I ran a simple Monte Carlo based test to compute the expected bias between a Poisson sample and the “equivalent” Gaussian sample. The result is shown in the plot below.

The jagged red line is the fractional expected bias relative to the true intensity. The typical recommendation in high-energy astronomy is to bin up events until there are about 25 or so counts per bin. This leads to an average bias of about 2% in the estimate of the true intensity. The bias drops below 1% for counts >50. Continue reading ‘Poisson vs Gaussian’ »

iFish in the archive

The iPhone App Store has a couple of apps that make life significantly easier for those of us inundated and overwhelmed by the stream of daily arXiv preprints. These are ArXivReader.app and ArXiv.app, both providing a means to browse and search the arXiv preprint database and both selling for 99c with the first selling for 99c and the second free. Check them out! The former even lets you save papers for off-line reading.

For me at least, the hardest part of going through the arXiv emails every day was to pick out the interesting papers in the deluge of text. These apps do the right thing and segregate the categories and highlight the titles. Fitts’ Law in action — suddenly the daily ritual is orders of magnitude more pleasant!

Correlation is not causation

What XKCD says:
xkcd on correlation: I used to think correlation implied causation - Then I took a statistics class.  Now I dont - Sounds like the class helped.  Well, maybe.

The mouseover text on the original says “Correlation doesn’t imply causation, but it does waggle its eyebrows suggestively and gesture furtively while mouthing ‘look over there’.”

It is a bad habit, hard to break, the temptation is great.

Lost in Translation: Measurement Error

You would think that something like “measurement error” is a well-defined concept, and everyone knows what it means. Not so. I have so far counted at least 3 different interpretations of what it means.

Suppose you have measurements X={Xi, i=1..N} of a quantity whose true value is, say, X0. One can then compute the mean and standard deviation of the measurements, E(X) and σX. One can also infer the value of a parameter θ(X), derive the posterior probability density p(θ|X), and obtain confidence intervals on it.

So here are the different interpretations:

  1. Measurement error is σX, or the spread in the measurements. Astronomers tend to use the term in this manner.
  2. Measurement error is X0-E(X), or the “error made when you make the measurement”, essentially what is left over beyond mere statistical variations. This is how statisticians seem to use it, essentially the bias term. To quote David van Dyk

    For us it is just English. If your measurement is different from the real value. So this is not the Poisson variability of the source for effects or ARF, RMF, etc. It would disappear if you had a perfect measuring device (e.g., telescope).

  3. Measurement error is the width of p(θ|X), i.e., the measurement error of the first type propagated through the analysis. Astronomers use this too to refer to measurement error.

Who am I to say which is right? But be aware of who you may be speaking with and be sure to clarify what you mean when you use the term!

MMIX

The year 2009 is the Darwin bicentennial and the sesquicentennial of the publication of the Origin of Species, but, um, even more importantly, it is the International Year of Astronomy, celebrating 400 orbits since Galileo started to look through a telescope.

“Thanks to Henrietta Leavitt”

[9/30/2008]

The CfA is celebrating the 100th anniversary of the discovery of the Cepheid period-luminosity relation on Nov 6, 2008. See http://www.cfa.harvard.edu/events/2008/leavitt/ for details.

[Update 10/03] For a nice introduction to the story of Henrietta Swan Leavitt, listen to this Perimeter Institute talk by George Johnson: http://pirsa.org/06050003/

[Update 11/06] The full program is now available. The symposium begins at Noon today.

Astroart Survey

Astronomy is known for its pretty pictures, but as Joe the Astronomer would say, those pretty pictures don’t make themselves. A lot of thought goes into maximizing scientific content while conveying just the right information, all discernible at a single glance. So the hardworkin folks at Chandra want your help in figuring out what works and how well, and they have set up a survey at http://astroart.cfa.harvard.edu/. Take the survey, it is both interesting and challenging!

Redistribution

RMF. It is a wørd to strike terror even into the hearts of the intrepid. It refers to the spread in the measured energy of an incoming photon, and even astronomers often stumble over what it is and what it contains. It essentially sets down the measurement error for registering the energy of a photon in the given instrument.

Thankfully, its usage is robustly built into analysis software such as Sherpa or XSPEC and most people don’t have to deal with the nitty gritty on a daily basis. But given the profusion of statistical software being written for astronomers, it is perhaps useful to go over what it means. Continue reading ‘Redistribution’ »

Whew

Contact has been re-established with XMM-Newton. Continue reading ‘Whew’ »

“planetariums and other foolishness”

Last month, Senator McCain (R-AZ) wildly dissed on Chicago’s Adler Planetarium, characterizing a funding request on its behalf as “planetariums and other foolishness.” Continue reading ‘“planetariums and other foolishness”’ »

Killer App

The iPhone is an amazing device. I have heard that some people use it as a phone, too, but it really is an extraordinary portable computer. It is faster and more powerful than the Sparcstations I used as a grad student, and will fit into your pocket. And most importantly, you can fit an entire planetarium on it.

There are many good planetarium programs that you can access on laptops, but it is really not that much fun to lug them around on camping trips or even out on to the roof at night. But now, thanks to the iPhone (and the iPod Touch) there has been a great leap forward. Continue reading ‘Killer App’ »

The Big Picture

Our hometown rag (the Boston Globe) runs an occasional series of photo collections that highlight news stories called The Big Picture. This week, they take a look at the Sun: http://www.boston.com/bigpicture/2008/10/the_sun.html

The pictures come from space and ground observatories, from SoHO, TRACE, Hinode, STEREO, etc. Goes without saying, the images are stunning, and some are even animated. The real kicker is that images such as these are being acquired by the hundreds, every hour upon the hour, 24/7/365.25 . It is like sipping from a firehose. Nobody can sit there and look at them all, so who knows what we are missing out on. Can statistics help? Can we automate a statistically robust “interestingness” criterion to filter the data stream that humans can then follow up on?

[Q] Objectivity and Frequentist Statistics

Is there an objective method to combine measurements of the same quantity obtained with different instruments?

Suppose you have a set of N1 measurements obtained with one detector, and another set of N2 measurements obtained with a second detector. And let’s say you wanted something as simple as an estimate of the mean of the quantity (say the intensity) being measured. Let us further stipulate that the measurement errors of each of the points is similar in magnitude and neither instrument displays any odd behavior. How does one combine the two datasets without appealing to subjective biases about the reliability or otherwise of the two instruments? Continue reading ‘[Q] Objectivity and Frequentist Statistics’ »

There and back again

The absolutely phenomenal webcomic XKCD hits a home run again, this time sketching out the spatial structure of the Universe all the way from here to The Edge .. in log scale. Continue reading ‘There and back again’ »

Blackbody Radiation [Eqn]

Like spherical cows, true blackbodies do not exist. Not because “black objects are dark, duh”, as I’ve heard many people mistakenly say — black here simply refers to the property of the object where no wavelength is preferentially absorbed or emitted, and all the energy input to it is converted into radiation. There are many famous astrophysical cases which are very good approximations to perfect blackbodies — the 2.73K microwave background radiation left over from the early Universe, for instance. Even the Sun is a good example. So it is often used to model the emission from various objects. Continue reading ‘Blackbody Radiation [Eqn]’ »