The AstroStat Slog

RMF. It is a wørd to strike terror even into the hearts of the intrepid. It refers to the spread in the measured energy of an incoming photon, and even astronomers often stumble over what it is and what it contains. It essentially sets down the measurement error for registering the energy of a photon in the given instrument.

Thankfully, its usage is robustly built into analysis software such as Sherpa or XSPEC and most people don’t have to deal with the nitty gritty on a daily basis. But given the profusion of statistical software being written for astronomers, it is perhaps useful to go over what it means. Continue reading ‘Redistribution’ »

In order to understand a learning procedure statistically it is necessary to identify two important aspects: its structural model and its error model. The former is most important since it determines the function space of the approximator, thereby characterizing the class of functions or hypothesis that can be accurately approximated with it. The error model specifies the distribution of random departures of sampled data from the structural model.

Continue reading ‘A Quote on Model’ »

People of experience would say very differently and wisely against what I’m going to discuss now. This post only combines two small cross sections of each branch of two trees, astronomy and statistics. Continue reading ‘survey and design of experiments’ »

Another deduced conclusion from reading preprints listed in arxiv/astro-ph is that astronomers tend to confuse classification and clustering and to mix up methodologies. They tend to think any algorithms from classification or clustering analysis serve their purpose since both analysis algorithms, no matter what, look like a black box. I mean a black box as in neural network, which is one of classification algorithms. Continue reading ‘Classification and Clustering’ »

A nice book by Christopher Bishop.
While I was reading abstracts and papers from astro-ph, I saw many applications of algorithms from pattern recognition and machine learning (PRML). The frequency will increase as large scale survey projects numerate, where recommending a good textbook or a reference in the field seems timely. Continue reading ‘[Book] pattern recognition and machine learning’ »

To claim results are powerful statistically, astronomers highly rely on eyeballing techniques (need apprenticeship to acquire skills but look subjective to me without such training). Some cases, I know actual statistical tests to support or to dissuade those claims. Hence, I believe astronomers are well aware of those statistical tests. I guess they are afraid that those statistics may reject their claims or are not powerful enough in numeric metrics. Instead, they spend efforts to make graphics more appealing. Continue reading ‘appealing eyes == powerful method’ »

The problem with data analysis is of course that it is a performing art. It is not something you easily write a paper on; rather, it is something you do. And so it is difficult to publish.

quoted from this conversation Continue reading ‘A Conversation with Peter Huber’ »

My greatest concern was what to call it. I thought of calling it “information”, but the word was overly used, so I decided to call it “uncertainty”. When I discussed it with John von Neumann, he had a better idea. Von Neumann told me, “You should call it entropy, for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that name, so it already has a name. In the second place, and more important, nobody knows what entropy really is, so in a debate you will always have the advantage.”

Continue reading ‘An anecdote on entrophy’ »

A question to astronomers. What do you think WLOG is? It has nothing to do with our BLOG or SLOG. Statisticians, please do not say a word. Continue reading ‘WLOG’ »

Like spherical cows, true blackbodies do not exist. Not because “black objects are dark, duh”, as I’ve heard many people mistakenly say — black here simply refers to the property of the object where no wavelength is preferentially absorbed or emitted, and all the energy input to it is converted into radiation. There are many famous astrophysical cases which are very good approximations to perfect blackbodies — the 2.73K microwave background radiation left over from the early Universe, for instance. Even the Sun is a good example. So it is often used to model the emission from various objects. Continue reading ‘Blackbody Radiation [Eqn]’ »

I still remember my first class as a new grad student. As a cocky Physics graduate, I was quite sure I knew plenty of astronomy. Astro 301, class 1, and it took all of 20 minutes of talk about stellar magnitudes to put that notion to permanent rest. So, for the sake of our stats colleagues, here’s a brief primer on one of the basic building blocks of astronomy. Continue reading ‘Magnitude [Eqn]’ »

Differential Emission Measures (DEMs) are a summary of the temperature structure of the outer atmospheres (aka coronae) of stars, and are usually derived from a select subset of line fluxes. They are notoriously difficult to estimate. Very few algorithms even bother to calculate error envelopes on them. They are also subject to numerous systematic uncertainties which can play havoc with proper interpretation. But they are nevertheless extremely useful since they allow changes in coronal structures to be easily discerned, and observations with one instrument can be used to derive these DEMs and these can then be used to predict what is observable with some other instrument. Continue reading ‘Differential Emission Measure [Eqn]’ »

I grew up in an environment that glamourized mathematical equations. Equations adorned a text like jewelry, set there to dazzle, and often to outshine the text that they were to illuminate. Needless to say, anything I wrote was dense, opaque, and didn’t communicate what it set out to. It was not until I saw a Reference Frame essay by David Mermin on how to write equations (1989, Physics Today, 42, p9) that I realized that equations should be treated as part of the text. You should be able to read them. David Mermin set out 3 rules for writing out equations, which I’ve tried to follow diligently (if not always successfully) since then. Continue reading ‘I Like Eq’ »

As mentioned before, background subtraction plays a big role in astrophysical analyses. For a variety of reasons, it is not a good idea to subtract out background counts from source counts, especially in the low-counts Poisson regime. What Bayesians recommend instead is to set up a model for the intensity of the source and the background and to infer these intensities given the data. Continue reading ‘Background Subtraction, the Sequel [Eqn]’ »

I have noticed that our statistician collaborators are often confused by our units. (Not a surprise; I, too, am constantly confused by our units.) One of the biggest culprits is the unit of energy, [keV], Continue reading ‘keV vs keV [Eqn]’ »