Jul 9th, 2008| 01:00 pm | Posted by vlk

The Kaplan-Meier (K-M) estimator is the non-parametric maximum likelihood estimator of the survival probability of items in a sample. “Survival” here is a historical holdover because this method was first developed to estimate patient survival chances in medicine, but in general it can be thought of as a form of cumulative probability. It is of great importance in astronomy because so much of our data are limited and this estimator provides an excellent way to estimate the fraction of objects that may be below (or above) certain flux levels. The application of K-M to astronomy was explored in depth in the mid-80′s by Jurgen Schmitt (1985, ApJ, 293, 178), Feigelson & Nelson (1985, ApJ 293, 192), and Isobe, Feigelson, & Nelson (1986, ApJ 306, 490). **[**See also Hyunsook's primer.**]** It has been coded up and is available for use as part of the ASURV package. Continue reading ‘Kaplan-Meier Estimator (Equation of the Week)’ »

Tags:

censored,

EotW,

Equation,

Equation of the Week,

Feigelson,

Isobe,

Kaplan-Meier,

maximum likelihood,

Nelson,

Schmitt,

survival analysis,

upper limit Category:

Frequentist,

Jargon,

Methods,

Stat |

13 Comments
Jul 8th, 2008| 07:27 pm | Posted by hlee

Astronomers confront with various censored and truncated data. Often these types of data are called after famous scientists who generalized them, like Eddington bias. When these censored or truncated data become the subject of study in statistics, instead of naming them, statisticians try to model them so that the uncertainty can be quantified. This area is called **survival analysis**. If your library has *The American Statistician* subscription and you are an astronomer handles censored or truncated data sets, this primer would be useful for briefly conceptualizing statistics jargon in survival analysis and for characterizing uncertainties residing in your data. Continue reading ‘Survival Analysis: A Primer’ »

Tags:

censored,

Efron,

Feigelson,

Freedman,

massive data,

Nelson,

Petrosian,

survival analysis,

truncated Category:

arXiv,

Fitting,

Stat |

4 Comments
Aug 14th, 2007| 10:17 pm | Posted by hlee

During the International X-ray Summer School, as a project presentation, I tried to explain the inadequate practice of χ^2 statistics in astronomy. *If your best fit is biased (any misidentification of a model easily causes such bias), do not use χ^2 statistics to get 1σ error for the 68% chance of capturing the true parameter.*

Later, I decided to do further investigation on that subject and this paper came along: Astrostatistics: Goodness-of-Fit and All That! by Babu and Feigelson.

Continue reading ‘Astrostatistics: Goodness-of-Fit and All That!’ »

Tags:

Anderson-Darling,

Babu,

best-fit,

bias,

bootstrap,

chi-square,

Cramer-von Mises,

Feigelson,

Kolmogorov-Smirnoff,

Kullback-Leibler distance,

nonparametric,

parametric,

resampling Category:

Algorithms,

arXiv,

Astro,

Fitting,

High-Energy,

Methods,

Spectral,

Stat |

7 Comments
Jan 29th, 2007| 12:34 am | Posted by hlee

Since Summer 2005, G. Jogesh Babu (Statistics) and Eric Feigelson (Astronomy) have organized lectures and lab sessions on statistics for astronomers and physicists. Lecturers are professors from Penn State statistics department and invited renown scientists from different countries. Students show diverse demography as well. Within a week or so, students listen Statistics 101 to recently published statistical theories particularly applied to astronomical data. They also learn how to use R, a statistical software and script language to perform statistics they learn through lectures. Past two years, this summer school proved its uniqueness and usefulness. More information on the upcoming school can be found at http://astrostatistics.psu.edu/su07/index.html and other topics regarding astrostatistics at Center for AstroStatistics at Penn State.