Jun 3rd, 2008| 02:53 am | Posted by vlk

It is somewhat surprising that astronomers haven’t cottoned on to Lowess curves yet. That’s probably a good thing because I think people already indulge in smoothing far too much for their own good, and Lowess makes for a very powerful hammer. But the fact that it is semi-parametric and is based on polynomial least-squares fitting does make it rather attractive.

And, of course, sometimes it is unavoidable, or so I told Brad W. When one has too many points for a regular polynomial fit, and they are too scattered for a spline, and too few to try a wavelet “denoising”, and no real theoretical expectation of any particular model function, and all one wants is “a smooth curve, damnit”, then Lowess is just the ticket.

Well, almost.

There is one major problem — *how does one figure what the error bounds are on the “best-fit” Lowess curve?* Clearly, each fit at each point can produce an estimate of the error, but simply collecting the separate errors is not the right thing to do because they would all be correlated. I know how to propagate Gaussian errors in boxcar smoothing a histogram, but this is a whole new level of complexity. Does anyone know if there is software that can calculate reliable error bands on the smooth curve? We will take any kind of error model — Gaussian, Poisson, even the (local) variances in the data themselves.

Tags:

Brad Wargelin,

error bands,

error bars,

Fitting,

least-squares,

Loess,

Lowess,

polynomial,

question for statisticians,

smoothing Category:

Algorithms,

Fitting,

Methods,

Stat,

Uncertainty |

11 Comments
Jan 25th, 2008| 12:53 pm | Posted by hlee

I have been observing some sorts of misconception about statistics and statistical nomenclature evolution in astronomy, which I believe, are attributed to the lack of references in the astronomical society. There are some textbooks designed for junior/senior science and engineering students, which are likely unknown to astronomers. Example-wise, these books are not suitable, to my knowledge. Although I never expect astronomers to learn standard graduate (mathematical) statistics textbooks, I do wish astronomers go beyond Numerical Recipes (W. H. Press, S. A. Teukolsky, W. T. Vetterling, & B. P. Flannery) and Error Data Reduction and Analysis for the Physical Sciences (P. R. Bevington & D. K. Robinson). Here are some good ones written by astronomers, engineers, and statisticians: Continue reading ‘Books – a boring title’ »

Tags:

application,

Bayesian,

book,

books,

box plot,

cartoon,

chi-square,

CLT,

correlation,

delta method,

EDA,

error bars,

Frequentist,

gaussian,

references,

scatter plot Category:

Cross-Cultural,

Data Processing,

Misc,

Stat |

12 Comments
Jan 21st, 2008| 03:33 pm | Posted by vlk

One of the big problems that has come up in recent years is in how to represent the uncertainty in certain estimates. Astronomers usually present errors as *+-stddev* on the quantities of interest, but that presupposes that the errors are uncorrelated. But suppose you are estimating a multi-dimensional set of parameters that may have large correlations amongst themselves? One such case is that of Differential Emission Measures (DEM), where the “quantity of emission” from a plasma (loosely, how much stuff there is available to emit — it is the product of the volume and the densities of electrons and H) is estimated for different temperatures. See the plots at the PoA DEM tutorial for examples of how we are currently trying to visualize the error bars. Another example is the correlated systematic uncertainties in effective areas (Drake et al., 2005, Chandra Cal Workshop). This is not dissimilar to the problem of determining the significance of a “feature” in an image (Connors, A. & van Dyk, D.A., 2007, SCMA IV). Continue reading ‘Dance of the Errors’ »

Tags:

animated,

David Garcia-Alvarez,

DEM,

error bands,

error bars,

flux,

MCMC,

O VII,

O VIII,

PINTofALE,

question for statisticians Category:

Algorithms,

Astro,

Data Processing,

Jargon,

MCMC,

Spectral,

Stars,

Uncertainty |

2 Comments