…in calculating the mean error of a series of observations it is preferable to use the simple mean residual irrespective of sign rather than the mean square residual

Such eminent astronomer said already least absolute deviation over chi-square, if I match simple mean residual and mean square residual to relevant methodologies, in order.

I guess there is a reason everything is done based on the chi-square although a small fraction of the astronomy community is aware of that the chi-square minimization is not the only utility function for finding best fits. The assumption that the residuals “(Observed – Expected)/sigma”, as written in chi-square methods, are (asymptotically) normal – Gaussian, is freely taken into account by astronomical data (astronomers who analyze these data) mainly because of their high volume. The worst case is that even if checking procedures for the Gaussianity are available from statistical literature, applying those procedures to astronomical data is either difficult or ignored. Anyway, if one is sure that the data/parameters of interest are sampled from normal distribution, Eddington’s statement is better to be reverted because of sufficiency. We also know the asymptotic efficiency of sample standard deviation when the probability density function satisfies more general regularity conditions than the Gaussian density.

As a statistician, it is easy to say, “assume that data are iid standard normal, wlog.” And then, develop a statistic, investigate its characteristics, and compare it with other statistics. If this statistics does not show promising results from the comparison and strictly suffers from this normality assumption, then statisticians will attempt to make this statistic robust by checking and relaxing assumptions and math. On the other hand, I’m not sure how much astronomers feel easy with this Gaussianity assumption in their data most of which are married to statistics or statistical tools based on the normal assumption. How often have the efforts of devising the statistic and trying different statistics been taken place?

Without imposing the Gaussianity assumption, I think that Eddington’s comment is extremely insightful. Commonly cited statistical methods in astronomy, like chi square methods, are built on Gaussianity assumption from which sample standard deviation is used for σ, the scale parameter of the normal distribution that is mapped to 68% coverage and multiple of the sample standard deviation correspond to well known percentiles as given in Numerical Recipes. In the end, I think statistical analysis in astronomy literature suffers from a dilemma, “which came first, the chicken or the egg?” On the other hand, I feel setback because such a insightful comment from one of the most renown astrophysicists didn’t gain much weight after many decades. My understanding that Eddington’s suggestion was ignored is acquired from reading only a fraction of publications in major astronomical journals; therefore, I might be wrong. Probably, astronomers use LAD and do robust inferences more often that I think.

Unless not sure about the Gaussianity in data (covering the sampling distribution, residuals between observed and expected, and some transformations), for inference problems, sample standard deviation may not be appropriate to get error bars with matching coverage. Estimating posterior distributions is a well received approach among some astronomers and there are good tutorials and textbooks about Bayesian data analysis for astronomers. Those familiar with basics of statistics, pyMC and its tutorial (or another link from python.org) will be very useful for proper statistical inference. If Bayesian computation sounds too cumbersome, for the simplicity, follow Eddington’s advice. Instead of sample standard deviation, use absolute mean deviation (simple mean residual, Eddington’s words) to quantify uncertainty. Perhaps, one wants to compare best fits and error bars from both strategies.

——————————————————
This story was inspired by Studies in the Hisotry of Probability and Statistics. XXXII: Laplace, Fisher, and the discovery of the concept of sufficiency by Stigler (1973) Biometrika v. 60(3), p.439. The quote of Eddington was adapted from this article. Another quote from this article I like to share:

Fisher went on to observe that this property of σ₂^[1] is quite dependent on the assumption that the population is normal, and showed that indeed σ₁^[2] is preferable to σ₂, at least in large samples, for estimating the scale parameter of the double exponential distribution, providing both estimators are appropriately rescaled

By assuming that each observations is normally (Gaussian) distributed with mean (mu) and variance (sigma^2), and that the object was to estimate sigma, Fisher proved that the sample standard deviation (or mean square residual) is more efficient than the mean deviation form the sample mean (or simple mean residual). Laplace proved it as well. The catch is that assumptions come first, not the sample standard deviation for estimating error (or sigma) of unknown distribution.

1. sample standard deviation
2. mean deviation from the sample mean

Every statistician, at least to my knowledge, knows E.L. Lehmann (his TPE and TSH are classic and Elements of Large Sample Theory was my textbook). Instead of reading daily astro-ph, I’m going through some collected papers from arxiv:stat and other references, in order to recover my hopefully existing research oriented mind (I like to share my stat or astrostat papers with you) and to continue slogging. The first one is an arxiv:math.ST paper by Lehmann.

His foremost knowledge and historic account on statistical models related to errors may benefit astronomers. Although I didn’t study history of astronomy and statistics, I’m very much aware of how astronomy innovated statistical thinking, particularly the area of large sample theory. Unfortunately, at the dawn of the 20th century, they went through an unwanted divorce. Papers from my [arXiv] series or a small portion of statistics papers citing astronomy, seem to pay high alimony without tax relief.

[math.ST:0805.2838] E.L.Lehmann
On the history and use of some standard statistical models

According to the author, the paper considers three assumptions: normality, independence, and the linear structure of the deterministic part. The particular reason for this paper into the slog is the following sentences:

The normal distribution became the acknowledged model for the distribution of errors of physical (particularly astronomical) measurements and was called the Law of Errors. It has a theoretical basis in the so called Law of Elementary Erros which assumed that an observational error is the sum of a large number of small independent errors and is therefore approximately normally distributed by the Central Limit Theorem.

A lot to be said but adding a quote referring Freedman that

“one problem noticeable to a statistician is that investigators do not pay attention to the stochastic assumptions behind the models. It does not seem possible to derive these assumptions from current theory, nor are they easily validated empirically on a case-by-case basis.”
The paper ends with the devastating conclusion:
“My opinion is that investigators need to think more about the underlying process, and look more closely at the data, without the distorting prism of convential (and largely irrelevant) stochastic models. Estimating nonexistent parameters cannot be very fruitful. And it must be equally a waste of time to test theories on the basis of statistical hypothesis that are rooted neither in prior theory nor in fact, even if the algorithms are recited in every statistics text without caveat.”

It is truly devastating.

A quote in the article referring the Preface of Snedecor’s book clearly tells the importance of collaborations.

“To the mathematical statistician must be delegated the task of developing the theory and devising the methods, accompanying these latter by adequate statements of the limitations of their use. …
None but the biologist can decide whether the conditions are fulfilled in his experiments.”

so does two sentences from the paper in the conclusion

A general text cannot provide the subject matter knowledge and the special features that are needed for successful modeling in specific cases. Experience with similar data is required, knowledge of theory and, as Freedman points out: shoe leather.

Other quotes in the article referring Scheffe,

“the effect of violation of the normality assumption is slightly on inferences about the mean but dangerous on inferences about variances.”

and Brownlee,

“applied statisticians have found empirically that usually there is no great need to fuss about the normality assumption”

and I confess that I’ve been fussing about astronomers’ gaussianity assumption; on the contrary, I advice my friends in other disciplines (for example, agriculture) treating their data with simpler analytic tools by assuming normality. To defend myself, I like to ask whether the independence assumption can be overlooked at the convenience of multiplying marginalized probabilities. I don’t think such concern/skepticism has not been addressed well enough compared to the normality assumption.

1. Gosset’s pen name was Student, from which the name, Student-t in t-distribution or t-test was spawned.