Bayesian Computation with R
Author:Jim Albert
Publisher: Springer (2007)

As the title said, accompanying R package LearnBayes is available (clicking the name will direct you for downloading the package). Furthermore, the last chapter is about WinBUGS. (Please, check out resources listed in BUGS for other BUGS, Bayesian inference Using Gibbs Sampling) Overall, it is quite practical and instructional. If an young astronomer likes to enter the competition posted below because of sophisticated data requiring non traditional statistical modeling, this book can be a good starting. (Here, traditional methods include brute force Monte Carlo simulations, chi^2/weighted least square fitting, and test statistics with rigid underlying assumptions).

An interesting quote is filtered because of a comment from an astronomer, “Bayesian is robust but frequentist is not” that I couldn’t agree with at the instance.

A Bayesian analysis is said to be robust to the choice of prior if the inference is insensitive to different priors that match the user’s beliefs.

Since there’s no discussion of priors in frequentist methods, Bayesian robustness cannot be matched and compared with frequentist’s robustness. Similar to my discussion in Robust Statistics, I kept the notion that robust statistics is insensitive to outliers or iid Gaussian model assumption. Particularly, the latter is almost ways assumed in astronomical data analysis, unless other models and probability densities are explicitly stated, like Poisson counts and Pareto distribution. New Bayesian algorithms are invented to achieve robustness, not limited to the choice of prior but covering the topics from frequentists’ robust statistics.

The introduction of Bayesian computation focuses on analytical and simple parametric models and well known probability densities. These models and their Bayesian analysis produce interpretable results. Gibbs sampler, Metropolis-Hasting algorithms, and their few hybrids could handle scientific problems as long as scientific models and the uncertainties both in observations and parameters transcribed into well known probability density functions. I think astronomers like to check Chap 6 (MCMC) and Chap 9 (Regression Models). Often times, in order to prove strong correlation between two variables, astronomers adopt simple linear regression models and fit the data to these models. A priori knowledge enhances the flexibility of fitting analysis in which Bayesian computation works robustly different from straightforward chi-square methods. The book does not have sophisticated algorithms nor theories. It only offers very necessities and foundations for Bayesian computations to be accommodated into scientific needs.

The other book is

Bayesian Core: A Practical Approach to Computational Bayesian Statistics.
Author: J. Marin and C.P.Robert
Publisher: Springer (2007).

Although the book is written by statisticians, the very first real data example is CMBdata (cosmic microwave background data; instead of cosmic, the book used cosmological. I’m not sure which one is correct but I’m so used to CMB by cosmic microwave background). Surprisingly, CMB became a very easy topic in statistics in terms of testing normality and extreme values. Seeing the real astronomy data first from the book was the primary reason of introducing this book. Also, it’s a relatively small volume book (about 250 pages) compared other Bayesian textbooks with the broad coverage of topics in Bayesian computation. There are other practical real data sets to illustrate Bayesian computations in the book and these example data sets are found from the book website

The book begins with R, then normal models, regression and variable selection, generalized linear models, capture-recapture experiments, mixture models, dynamic models, and image analysis are covered.

I feel exuberant when I found the book describes the law of large numbers (LLN) that justifies the Monte Carlo methods. The LLN appears often when integration is approximated by summation, which astronomers use a lot without referring the name of this law. For more information, I rather give a wikipedia link to Law of Large Numbers.

Several MCMC algorithms can be mixed together within a single algorithm using either a circular or a random design. While this construction is often suboptimal (in that the inefficient algorithms in the mixture are still used on a regular basis), it almost always brings an improvement compared with its individual components. A special case where a mixed scenario is used is the Metropolis-within-Gibbs algorithm: When building a Gibbs sample, it may happen that it is difficult or impossible to simulate from some of the conditional distributions. In that case, a single Metropolis step associated with this conditional distribution (as its target) can be used instead.

Description in Sec. 4.2 Metropolis-Hasting Algorithms is expected to be more appreciated and comprehended by astronomers because of the historical origins of these topics, detailed balance equation and random walk.

Personal favorite is section 6 on mixture models. Astronomers handle data of multi populations (multiple epochs of star formations, single or multiple break power laws, linear or quadratic models, metalicities from merging or formation triggers, backgrounds+sources, environment dependent point spread functions, and so on) and discusses the difficulties of label switching problems (identifiability issue in codifying data into MCMC or EM algorithm)

A completely different approach to the interpretation and estimation of mixtures is the semiparametric perspective. To summarize this approach, consider that since very few phenomena obey probability laws corresponding to the most standard distributions, mixtures such as (*) can be seen as a good trade-off between fair represntation of the phenomenon and efficient estimation of the underlying distribution. If k is large enough, there is theoretical support for the argument that (*) provides a good approximation (in some functional sense) to most distributions. Hence, a mixture distribution can be perceived as a type of basis approximation of unknown distributions, in a spirit similar to wavelets and splines, but with a more intuitive flavor (for a statistician at least). This chapter mostly focuses on the “parametric” case, when the partition of the sample into subsamples with different distributions f_j does make sense form the dataset point view (even though the computational processing is the same in both cases).

We must point at this stage that mixture modeling is often used in image smoothing but not in feature recognition, which requires spatial coherence and thus more complicated models…

My patience ran out to comprehend every detail of the book but the section of reversible jump MCMC, hidden Markov model (HMM), and Markov random fields (MRF) would be very useful. Particularly, these topics appear often in image processing, which field astronomers have their own algorithms. Adaption and comparison across image analysis methods promises new directions of scientific imaging data analysis beyond subjective denoising, smoothing, and segmentation.

Readers considering more advanced Bayesian computation and rigorous treatment of MCMC methodology, I’d like to point a textbook, frequently mentioned by Marin and Robert.

Monte Carlo Statistical Methods Robert, C. and Casella, G. (2004)
Springer-Verlag, New York, 2nd Ed.

There are a few more practical and introductory Bayesian Analysis books recently published or soon to be published. Some readership would prefer these books of running ink. Perhaps, there is/will be Bayesian Computation with Python, IDL, Matlab, Java, or C/C++ for those who never intend to use R. By the way, for Mathematica users, you would like to check out Phil Gregory’s book which I introduced in [books] a boring title. My point is that applied statistics has become more friendly to non statisticians through these good introductory books and free online materials. I hope more astronomers apply statistical models in their data analysis without much trouble in executing Bayesian methods. Some might want to check BUGS, introduced [BUGS]. This posting contains resources of how to use BUGS and available packages under languages.

There are three distinctive subjects in spatial statistics: geostatistics, lattice data analysis, and spatial point pattern analysis. Because of the resemblance between the spatial distribution of observations in coordinates and the notion of spatially random points, spatial statistics in astronomy has leaned more toward the spatial point pattern analysis than the other subjects. In other fields from immunology to forestry to geology whose data are associated spatial coordinates of underlying geometric structures or whose data were sampled from lattices, observations depend on these spatial structures and scientists enjoy various applications from geostatistics and lattice data analysis. Particularly, kriging is the fundamental notion in geostatistics whose application is found many fields.

Hitherto, I expected that the term kriging can be found rather frequently in analyzing cosmic micro-wave background (CMB) data or large extended sources, wide enough to assign some statistical models for understanding the expected geometric structure and its uncertainty (or interpolating observations via BLUP, best linear unbiased prediction). Against my anticipation, only one referred paper from ADS emerged:

Topography of the Galactic disk – Z-structure and large-scale star formation
by Alfaro, E. J., Cabrera-Cano, J., and Delgado (1991)
in ApJ, 378, pp. 106-118

I attribute this shortage of applying kriging in astronomy to missing data and differential exposure time across the sky. Both require underlying modeling to fill the gap or to convolve with observed data to compensate this unequal sky coverage. Traditionally the kriging analysis is only applied to localized geological areas where missing and unequal coverage is no concern. As many survey and probing missions describe the wide sky coverage, we always see some gaps and selection biases in telescope pointing directions. So, once this characteristics of missing is understood and incorporated into models of spatial statistics, I believe statistical methods for spatial data could reveal more information of our Galaxy and universe.

A good news for astronomers is that nowadays more statisticians and geo-scientists working on spatial data, particularly from satellites. These data are not much different compared to traditional astronomical data except the direction to which a satellite aims (inward or outward). Therefore, data of these scientists has typical properties of astronomical data: missing, unequal sky coverage or exposure and sparse but gigantic images. Due to the increment of computational power and the developments in hierarchical modeling, techniques in geostatistics are being developed to handle these massive, but sparse images for statistical inference. Not only denoising images but they also aim to produce a measure of uncertainty associated with complex spatial data.

For those who are interested in what spatial statistics does, there are a few books I’d like to recommend.

• Cressie, N (1993) Statistics for spatial data
  (the bible of statistical statistics)
• Stein, M.L. (2002) Interpolation of Spatial Data: Some Theory for Kriging
  (it’s about Kriging and written by one of scholarly pinnacles in spatial statistics)
• Banerjee, Carlin, and Gelfand (2004) Hierarchical Modeling and Analysis for Spatial Data
  (Bayesian hierarchical modeling is explained. Very pragmatic but could give an impression that it’s somewhat limited for applications in astronomy)
• Illian et al (2008) Statistical Analysis and Modelling of Spatial Point Patterns
  (Well, I still think spatial point pattern analysis is more dominant in astronomy than geostatistics. So… I feel obliged to throw a book for that. If so, I must mention Peter Diggle’s books too.)
• Diggle (2004) Statistical Analysis of Spatial Point Patterns
  Diggle and Ribeiro (2007) Model-based Geostatistics
  

• [astro-ph:0806.2228] T.J. Cornwell
     Multi-Scale CLEAN deconvolution of radio synthesis images
• [astro-ph:0806.2575] C.A.L. Bailer-Jones
     What will Gaia tell us about the Galactic disk?
• [astro-ph:0806.2823] Williams et al.
     Lensed Image Angles: New Statistical Evidence for Substructure (Apart from their K-S tests, personally lensing is considered to be a nice subject from a geostatistics standpoint.)
• [astro-ph:0806.3074] Eriksen and Wehus
     Marginal distributions for cosmic variance limited CMB polarization data
• [astro-ph:0806.2969] W. Boschin et al.
     Optical analysis of the poor clusters Abell 610, Abell 725, and Abell 796, containing diffuse radio sources (astronomers call gaussian mixture models by KMM)
• [astro-ph:0806.3096] Miller, Shimon, & Keating
     CMB Beam Systematics: Impact on Lensing Parameter Estimation (Note Monte Carlo Markov Chain in the abstract, not Markov chain Monte Carlo.)

• [astro-ph:0806.0650] Kimball and Ivezi\’c
  A Unified Catalog of Radio Objects Detected by NVSS, FIRST, WENSS, GB6, and SDSS (The catalog is available HERE. I’m always fascinated with the possibilities in catalog data sets which machine learning and statistics can explore. And I do hope that the measurement error columns get recognition from non astronomers.)

• [astro-ph:0806.0820] Landau and Simeone
  A statistical analysis of the data of Delta \alpha/ alpha from quasar absorption systems (It discusses Student t-tests from which confidence intervals for unknown variances and sample size based on Type I and II errors are obtained.)

• [stat.ML:0806.0729] R. Girard
  High dimensional gaussian classification (Model based – gaussian mixture approach – classification, although it is often mentioned as clustering in astronomy, on multi- dimensional data is very popular in astronomy)

• [astro-ph:0806.0520] Vio and Andreani
  A Statistical Analysis of the “Internal Linear Combination” Method in Problems of Signal Separation as in CMB Observations (Independent component analysis, ICA is discussed)

• [astro-ph:0806.0560] Nobel and Nowak
  Beyond XSPEC: Towards Highly Configurable Analysis (The flow of spectral analysis with XSPEC and Sherpa has not been accepted smoothly; instead, it has been a personal struggle. It seems the paper considers XSPEC as a black box, which I completely agree with. The main objective of the paper is comparing XSPEC and ISIS)

• [astro-ph:0806.0113] Casandjian and Grenier
  A revised catalogue of EGRET gamma-ray sources (The maximum likelihood detection method, which I never heard from statistical literature, is utilized)

• [astro-ph:0805.1290]R. Barnard, L. Shaw Greening, U. Kolb
  A multi-coloured survey of NGC 253 with XMM-Newton: testing the methods used for creating luminosity functions from low-count data

• [astro-ph:0805.1469] Philip J. Marshall et al.
  Automated detection of galaxy-scale gravitational lenses in high resolution imaging data

• [astro-ph:0805.1470] E. P. Kontar, E. Dickson, J. Kasparova
  Low-energy cutoffs and in electron spectra of solar flares: statistical survey (It is not statistically rigorous but the topic can be connected to dip tests or gap tests in statistics)

• [astro-ph:0805.1936] J. Yee & B. Gaudi
  Characterizing Long-Period Transiting Planets Observed by Kepler (discusses uncertainty in light curves and Fisher matrix)

• [astro-ph:0805.1994] the QUad collaboration: C. Pryke et al.
  Second and third season QUaD CMB temperature and polarization power spectra (What is jackknife tests? A brief scan of the paper does not register with my understanding of jackknifing. It looks more close to cross validation. Another slog topic shall come: bootstrap, cross validation, jackknife, and resampling.)

• [astro-ph:0805.2121] N. Cole et al.
  Maximum Likelihood Fitting of Tidal Streams With Application to the Sagittarius Dwarf Tidal Tails

• [astro-ph:0805.2155] J Yoo & M Zaldarriaga
  Improved estimation of cluster mass profiles from the cosmic microwave background

• [astro-ph:0805.2207] A.Vikhlinin et al.
  Chandra Cluster Cosmology Project II: Samples and X-ray Data Reduction (it mentions calibration uncertainty and background, can it be a reference to stacking, coadding, source detection, etc?)

• [astro-ph:0805.2325] J.M. Loh
  A valid and fast spatial bootstrap for correlation functions

• [astro-ph:0805.2326] T. Wickramasinghe, M. Struble, J. Nieusma
  Observed Bimodality of the Einstein Crossing Times of Galactic Microlensing Events

• [astro-ph:0804.2904]M. Cruz et al.
  The CMB cold spot: texture, cluster or void?

• [astro-ph:0804.2917] Z. Zhu, M. Sereno
  Testing the DGP model with gravitational lensing statistics

• [astro-ph:0804.3390] Valkenburg, Krauss, & Hamann
  Effects of Prior Assumptions on Bayesian Estimates of Inflation Parameters, and the expected Gravitational Waves Signal from Inflation

• [astro-ph:0804.3413] N.Ball et al.
  Robust Machine Learning Applied to Astronomical Datasets III: Probabilistic Photometric Redshifts for Galaxies and Quasars in the SDSS and GALEX (Another related publication [astro-ph:0804.3417])

• [astro-ph:0804.3471] M. Cirasuolo et al.
  A new measurement of the evolving near-infrared galaxy luminosity function out to z~4: a continuing challenge to theoretical models of galaxy formation

• [astro-ph:0804.3475] A.D. Mackey et al.
  Multiple stellar populations in three rich Large Magellanic Cloud star clusters

• [stat.ME:0804.3853] C. R\”over , R. Meyer, N. Christensen
  Modelling coloured noise (MCMC & sunspot data)

• [astro-ph:0804.1809] H. Khiabanian, I.P. Dell’Antonio
  A Multi-Resolution Weak Lensing Mass Reconstruction Method (Maximum likelihood approach; my naive eyes sensed a certain degree of relationship to the GREAT08 CHALLENGE)

• [astro-ph:0804.1909] A. Leccardi and S. Molendi
  Radial temperature profiles for a large sample of galaxy clusters observed with XMM-Newton

• [astro-ph:0804.1964] C. Young & P. Gallagher
  Multiscale Edge Detection in the Corona

• [astro-ph:0804.2387] C. Destri, H. J. de Vega, N. G. Sanchez
  The CMB Quadrupole depression produced by early fast-roll inflation: MCMC analysis of WMAP and SDSS data

• [astro-ph:0804.2437] P. Bielewicz, A. Riazuelo
  The study of topology of the universe using multipole vectors

• [astro-ph:0804.2494] S. Bhattacharya, A. Kosowsky
  Systematic Errors in Sunyaev-Zeldovich Surveys of Galaxy Cluster Velocities

• [astro-ph:0804.2631] M. J. Mortonson, W. Hu
  Reionization constraints from five-year WMAP data

• [astro-ph:0804.2645] R. Stompor et al.
  Maximum Likelihood algorithm for parametric component separation in CMB experiments (separate section for calibration errors)

• [astro-ph:0804.2671] Peeples, Pogge, and Stanek
  Outliers from the Mass–Metallicity Relation I: A Sample of Metal-Rich Dwarf Galaxies from SDSS

• [astro-ph:0804.2716] H. Moradi, P.S. Cally
  Time-Distance Modelling In A Simulated Sunspot Atmosphere (discusses systematic uncertainty)

• [astro-ph:0804.2761] S. Iguchi, T. Okuda
  The FFX Correlator

• [astro-ph:0804.2742] M Bazarghan
  Automated Classification of ELODIE Stellar Spectral Library Using Probabilistic Artificial Neural Networks

• [astro-ph:0804.2827]S.H. Suyu et al.
  Dissecting the Gravitational Lens B1608+656: Lens Potential Reconstruction (Bayesian)

• [astro-ph:0802.3411] N.Seto
     Detecting Planets around Compact Binaries with Gravitational Wave Detectors in Space

• [astro-ph:0802.3599] Jones & van de Weygaert
     Cosmic Order out of Primordial Chaos: a tribute to Nikos Voglis

• [astro-ph:0802.3764] J. Southworth
     Homogeneous studies of transiting extrasolar planets. I. Light curve analyses

• [astro-ph:0802.3914] A. M. Wolfe et.al.
      Bimodality in Damped Lyman alpha Systems (Kolmogorov-Smirnoff test)

• [astro-ph:0802.3931] M.S. Wheatland
     The Energetics of a Flaring Solar Active Region, and Observed Flare Statistics

A paper linked with an animation: [astro-ph:0802.3664] Dubinski and Farah
    GRAVITAS: Portraits of a Universe in Motion (I wanted to but I couldn’t watch.)
Lecture notes on CMB: [astro-ph:0802.3688] Wayne Hu
    Lecture Notes on CMB Theory: From Nucleosynthesis to Recombination

A review paper on LASSO and LAR: [stat.ME:0801.0964] T. Hesterberg et.al.
   Least Angle and L₁ Regression: A Review
Model checking for Bayesian hierarchical modeling: [stat.ME:0802.0743] M. J. Bayarri, M. E. Castellanos
   Bayesian Checking of the Second Levels of Hierarchical Models

• [astro-ph:0802.0042] Y. Kubo
  Statistical Models for Solar Flare Interval Distribution in Individual Active Regions (it discusses AIC)

• [astro-ph:0802.0131] J.Bobin, J-L Starck and R. Ottensamer
  Compressed Sensing in Astronomy

• [astro-ph:0802.0387] J. Gaite
  Geometry and scaling of cosmic voids

• [astro-ph:0802.0400] R. Vio & P. Andreani
  A Modified ICA Approach for Signal Separation in CMB Maps

• [astro-ph:0802.0498] V. Balasubramanian, K. Larjo and R. Sheth
  Experimental design and model selection: The example of exoplanet detection

• [astro-ph:0802.0537] G. Dan, Z. Yanxia, & Z. Yongheng
  Support Vector Machines and Kd-tree for Separating Quasars from Large Survey Databases

To find the paper or other interesting ones, click

• [astro-ph:0801.0638]
  AIC, BIC, Bayesian evidence and a notion on simplicity of cosmological model M Szydlowski & A. Kurek

• [astro-ph:0801.0642]
  Correlation of CMB with large-scale structure: I. ISW Tomography and Cosmological Implications S. Ho et.al.

• [astro-ph:0801.0780]
  The Distance of GRB is Independent from the Redshift F. Song

• [astro-ph:0801.1081]
  A robust statistical estimation of the basic parameters of single stellar populations. I. Method X. Hernandez and D. Valls–Gabaud

• [astro-ph:0801.1106]
  A Catalog of Local E+A(post-starburst) Galaxies selected from the Sloan Digital Sky Survey Data Release 5 T. Goto (Carefully built catalogs are wonderful sources for classification/supervised learning, or semi-supervised learning)

• [astro-ph:0801.1358]
  A test of the Poincare dodecahedral space topology hypothesis with the WMAP CMB data B.S. Lew & B.F. Roukema

In cosmology, a few candidate models to be chosen, are generally nested. A larger model usually is with extra terms than smaller ones. How to define the penalty for the extra terms will lead to a different choice of model selection criteria. However, astronomy papers in general never discuss the consistency or statistical optimality of these selection criteria; most likely Monte Carlo simulations and extensive comparison across those criteria. Nonetheless, my personal thought is that the field of model selection should be encouraged to astronomers to prevent fallacies of blindly fitting models which might be irrelevant to the information that the data set contains. Physics tells a correct model but data do the same.