The AstroStat Slog » application http://hea-www.harvard.edu/AstroStat/slog Weaving together Astronomy+Statistics+Computer Science+Engineering+Intrumentation, far beyond the growing borders Fri, 09 Sep 2011 17:05:33 +0000 en-US hourly 1 http://wordpress.org/?v=3.4 [ArXiv] Special Issue from Annals of Applied Statistics http://hea-www.harvard.edu/AstroStat/slog/2009/arxiv-special-issue-from-annals-of-applied-statistics/ http://hea-www.harvard.edu/AstroStat/slog/2009/arxiv-special-issue-from-annals-of-applied-statistics/#comments Mon, 09 Feb 2009 10:02:01 +0000 hlee http://hea-www.harvard.edu/AstroStat/slog/?p=1520 When I was studying astronomy, during when I once became a subject for a social science survey study about life in a department where gender bias is extreme (I was only female), people often asked me how to forecast weather or how to predict future (boys often get questions related to becoming astronauts in addition to weather men and astrologists). Relating astronomy to earth science still happens. Statisticians that I met at conferences, often tried to associate my efforts on astronomical data with those of geologists and meteorologists, who often use stochastic models and spatial temporal models, dimensional extensions of models in time series. Because of this confusion between astronomy and meteorology/geology/oceanology, and the longer history of wide statistical applications found from the latter subjects (a good counter example is the least square method by Gauss but I cannot think more examples to contradict my statement that statistics is used widely among earth scientists with rich history), from time to time my attention has been paid to various applications and models in those subjects so as to find a thread for similar applications for astronomy. Although I don’t like the misconception of astronomy equal to meteorology or geoscience, those scientific fields, what so ever, share at least one commonality that statistical methods are applied to analyzing satellite data.

There is a special issue on Atmospheric Science from the Annals of Applied Statistics, offering me intriguing discussions for finding a common ground between atmospheric science and astronomy. If the general public perception cannot tell the difference between meteorology and astronomy, despite the fact that my affirmative reply to statisticians’ comments on my interests in astronomy always has been “Astronomy and meteorology are very different scientific disciplines,” let’s find out some similarities from how statistics is applied. Astronomers can find more useful applications in the issue from their ends. Here, provided are some interesting ones from my judegment with their [arXiv] links. The whole issue’s table of contents given here: AoAS, vol 2, issue 4 (2008). Most of articles are now to be located at arXiv.

[arxiv:0901.3665] Parameter estimation for computationally intensive nonlinear regression with an application to climate modeling
by D. Drignei, C. E. Forest, and D. Nychka
:I wish for your attention to the sections about constructing a surrogate for the nonlinear complex climate model
[arxiv:0901.3670] Interpolating fields of carbon monoxide data using a hybrid statistical-physical model
by A. Malmberg, A. Arellano, D. P. Edwards, N. Flyer, D. Nychka, C. Wikle
:many astronomers would find more similarities in approaches by reading the abstract than I would. The only difference would be that they are using Carbon Oxide data as a result of the earth green house effect
[arxiv:0901.3494] Interpreting self-organizing maps through space–time data models
by H. Sang, A. E. Gelfand, C. Lennard, G. Hegerl, B. Hewitson
:a good reference for astronomers interested in SOM for high dimensional data and dimension reduction

]]>
http://hea-www.harvard.edu/AstroStat/slog/2009/arxiv-special-issue-from-annals-of-applied-statistics/feed/ 1
Books – a boring title http://hea-www.harvard.edu/AstroStat/slog/2008/books-a-boring-title/ http://hea-www.harvard.edu/AstroStat/slog/2008/books-a-boring-title/#comments Fri, 25 Jan 2008 16:53:21 +0000 hlee http://hea-www.harvard.edu/AstroStat/slog/2008/books-a-boring-title/ I have been observing some sorts of misconception about statistics and statistical nomenclature evolution in astronomy, which I believe, are attributed to the lack of references in the astronomical society. There are some textbooks designed for junior/senior science and engineering students, which are likely unknown to astronomers. Example-wise, these books are not suitable, to my knowledge. Although I never expect astronomers to learn standard graduate (mathematical) statistics textbooks, I do wish astronomers go beyond Numerical Recipes (W. H. Press, S. A. Teukolsky, W. T. Vetterling, & B. P. Flannery) and Error Data Reduction and Analysis for the Physical Sciences (P. R. Bevington & D. K. Robinson). Here are some good ones written by astronomers, engineers, and statisticians:

The motivation of writing this posting was originated to Vinay’s recommendation: Practical Statistics for Astronomers (J.V.Wall and C.R.Jenkins), which provides many statistical insights and caveats that astronomers tend to ignore. Without looking at the error distribution and the properties of data, astronomers jump into chi-square and correlation. If someone reads the book, he/she will be careful on adopting statistics of common practice in astronomy, developed many decades ago, and founded on strong assumptions, not compatible with modern data sets. The book addresses many concerns that have been growing in my mind for astronomers and introduces various statistical methods applicable in astronomy.

The view points of astronomers without in-class statistics education but with full readership of this book, would be different from mine. The book mentioned unbiasedness, consistency, closedness, and robustness of statistics, which normally are not discussed nor proved in astronomy papers. Therefore, those readers may miss the insights, caveats, and contents-between-the-lines of the book, which I care about. To reduce such gap, as for quick and easy understanding of classical statistics, I recommend Cartoon Guide to Statistics (Larry Gonick, Woollcott Smith Business & Investing Collins) as a first step. This cartoon book enhances fundamentals in statistics only with fun and a friendly manner, and provides everything that rudimentary textbooks offer.

If someone wants to know beyond classical statistics (so called frequentist statistics) and likes to know popular Bayesian statistics, astronomy professor Phil Gregory’s Bayesian Logical Data Analysis for the Physical Sciences is recommended. If one likes to know little bit more on the modern statistics of frequentists and Bayesians, All of Statistics (Larry Wasserman) is recommended. I realize that textbooks for non-statistics students are too thick to go through in a short time (The book for senior engineering students at Penn State I used for teaching was Probability and Statistics for Engineering and the Sciences by Jay. L Devore, 4th and 5th edition and it was about 600 pages. The current edition is 736 pages). One of well received textbooks for graduate students in electrical engineering is Probability, Random Variables and Stochastic Processes (A. Papoulis & S.U. Pillai). I remember the book offers a rather less abstract definition of measure and practical examples (Personally, Hermite polynomials was useful from the book).

For a casual reading about statistics and its 20th century history, The Lady Tasting Tea: How Statistics Revolutionized Science in the Twentieth Century (D. Salsburg) is quite nice.

Statistics is not just for best fit analysis and error bars. It is a wonderful telescope extracts correct information when it is operated carefully to the right target by the manual. It gets rid of atmospheric and other blurring factors when statistics is understood righteously. It is not a black box nor a magic, as many people think.

The era of treating everything gaussian is over decades ago. Because of the central limit theorem and the delta method (a good example is log-transformation), many statistics asymptotically follows the normal (gaussian) distribution but there are various families of distributions. Because of possible bias in the chi-square method, the error bar cannot guarantee the appointed coverage, like 95%. There are also nonparametric statistics, known for robustness, whereas it may be less efficient than statistics of distribution family assumption. Yet, it does not require model assumption. Also, Bayesian statistics works wonderfully if correct information on priors, suitable likelihood models, and computing powers for hierarchical models and numerical integration are provided.

Before jumping into the chi-square for fitting and testing at the same time, to prevent introducing bias, exploratory data analysis is required for better understanding data and for seeking a suitable statistic and its assumptions. The exploratory data analysis starts from simple scatter plots and box plots. A little statistical care for data and good interests in the truth of statistical methods are all I am asking for. I do wish that these books could assist the realization of my wishes.

—————————————————————————-
[1.] Most of links to books are from amazon.com but there is no personal affiliation to the company.

[2.] In addition to the previous posting on chi-square, what is so special about chi square in astronomy, I’d like to mention possible bias in chi-square fitting and testing. It is well known that utilizing the same data set for fitting, which results in parameter estimates so called in astronomy best fit values and error bars, and testing based on these parameter estimates brings out bias so that the best fit is biased from the true parameter value and the error bar does not match the aimed coverage. See the problem from Aneta’s an example of chi2 bias in fitting x-ray spectra

[3.] More book recommendation is welcome.

]]>
http://hea-www.harvard.edu/AstroStat/slog/2008/books-a-boring-title/feed/ 12
[ArXiv] Random Matrix, July 13, 2007 http://hea-www.harvard.edu/AstroStat/slog/2007/arxiv-random-matrix-july-13-2007/ http://hea-www.harvard.edu/AstroStat/slog/2007/arxiv-random-matrix-july-13-2007/#comments Mon, 16 Jul 2007 17:30:23 +0000 hlee http://hea-www.harvard.edu/AstroStat/slog/2007/arxiv-random-matrix-july-13-2007/ From arxiv/astro-ph:0707.1982v1,
Nflation: observable predictions from the random matrix mass spectrum by Kim and Liddle

To my knowledge, random matrix received statisticians’ interests fairly recently and SAMSI (Statistical and Applied Mathematical Sciences Institute) offered a semester long program on High Dimensional Inference and Random Matrices (tutorials and lecture notes can be found) during Fall 2006 . However, my knowledge is very limited to make a comment or critic on Kim and Liddle’s paper. Clearly, nonetheless, this paper is not about random matrix theory but about its straightforward application to the cosmological model viability.

A. Liddle has published papers on theoretic cosmology from a statistical model based approach (the ones I’ve seen are most likely related to statistical model selection). Personally, I like his book: An Introduction to Modern Cosmology (2nd ed. ISBN 0-470-84835-9), which might be useful to statisticians who wish to work with cosmologists.

]]>
http://hea-www.harvard.edu/AstroStat/slog/2007/arxiv-random-matrix-july-13-2007/feed/ 0