Mar 31st, 2009| 03:43 pm | Posted by hlee

Before using any adaptations of chi-square statistic, please spend a minute or two to ponder whether your strategy with chi-square belongs one of these categories.

**1**. Lack of independence among the single events or measures

**2**. Small theoretical frequencies

**3**. Neglect of frequencies of non-occurrence

**4**. Failure to equalize \sum O_i (the sum of the observed frequencies) and \sum M_i (the sum of the theoretical frequencies)

**5**. Indeterminate theoretical frequencies

**6**. Incorrect or questionable categorizing

**7**. Use of non-frequency data

**8**. Incorrect determination of the number of degrees of freedom

**9**. Incorrect computations (including a failure to weight by N when proportions instead of frequencies are used in the calculations)

From “**Chapter 10: On the Use and Misuse of Chi-square**” by K.L.Delucchi in *A Handbook for Data Analysis in the Behavioral Sciences* (1993). Delucchi acknowledged these nine principle sources of error to Lewis and Burke (1949), entitled “The Use and Misuse of the Chi-square” published in *Psychological Bulletin.* Continue reading ‘Use and Misuse of Chi-square’ »

Mar 25th, 2009| 10:13 am | Posted by chasc

From Christian Fendt comes this announcement:

——————————————————————

First Announcement and Call for Applications

——————————————————————

The “International Max Planck Research School for Astronomy & Cosmic Physics at the University of Heidelberg” (IMPRS-HD)

announces the

— 4th Heidelberg Summer School:

— Statistical Inferences from Astrophysical Data

— August 10-14, 2009

Continue reading ‘[Announce] Heidelberg Summer School’ »

Tags:

2009,

Announcement,

AstroStat,

August,

Heidelberg,

IMPRS,

IMPRS-HD,

inference,

Max Planck,

summer school Category:

Astro,

Misc,

News,

Stat |

Comment
Mar 24th, 2009| 03:18 pm | Posted by chasc

From Jogesh Babu comes this announcement:

Summer School in Statistics for Astronomers V

June 1-6, 2009

Penn State University

http://astrostatistics.psu.edu/su09/

Continue reading ‘[Announce] AstroStat Summer School at Penn State’ »

Mar 18th, 2009| 08:30 pm | Posted by hlee

I was disappointed when video, audio, or handout files were not available from the research program “Statistical Theory and Methods for Complex High-Dimensional Data” held at Isaac Newton Institute for Mathematical Sciences during the first half of last year after checking the sites several times. Wow…They are now there~ Continue reading ‘Web Seminar’ »

Mar 17th, 2009| 03:37 pm | Posted by hlee

I couldn’t believe my eyes when I saw 4754 degrees of freedom (d.f.) and chi-square test statistic 4859. I’ve often enough seen large degrees of freedom from journals in astronomy, several hundreds to a few thousands, but I never felt comfortable at these big numbers. Then with a great shock 4754 d.f. appeared. I must find out why I feel so bothered at these huge degrees of freedom. Continue reading ‘4754 d.f.’ »

Tags:

Binning,

chi-square,

chi-square minimization,

chi-square optimization,

chi-square statistic,

class,

degrees-of-freedom,

equiprobable,

goodness-of-fit test,

kernel density estimation Category:

Bad AstroStat,

Fitting,

High-Energy,

Methods,

Spectral,

X-ray |

2 Comments
Mar 11th, 2009| 01:04 pm | Posted by hlee

by **T. Cover and J. Thomas** website: http://www.elementsofinformationtheory.com/

Once, perhaps more, I mentioned this book in my post with the most celebrated paper by Shannon (see the posting). Some additional recommendation of the book has been made to answer offline inquiries. And this book always has been in my favorite book list that I like to use for teaching. So, I’m not shy with recommending this book to astronomers with modern objective perspectives and practicality. Before advancing for more praises, I must say that those admiring words do not imply that I understand every line and problem of the book. Like many fields, Information theory has grown fast since the monumental debut paper by Shannon (1948) like the speed of astronomers observation techniques. Without the contents of this book, most of which came after Shannon (1948), internet, wireless communication, compression, etc could not have been conceived. Since the notion of “**entropy**“, the core of **information theory**, is familiar to astronomers (physicists), the book would be received better among them than statisticians. This book should be read easier to astronomers than statisticians. Continue reading ‘[Book] Elements of Information Theory’ »

Tags:

bandwidth,

book,

Cover,

data mining,

education,

Entropy,

Information theory,

Kolmogorov complexity,

Shannon,

Thomas Category:

Algorithms,

arXiv,

Cross-Cultural,

Data Processing,

Jargon,

Quotes |

Comment
Mar 10th, 2009| 12:41 pm | Posted by vlk

The iPhone App Store has a couple of apps that make life significantly easier for those of us inundated and overwhelmed by the stream of daily arXiv preprints. These are ArXivReader.app and ArXiv.app, both providing a means to browse and search the arXiv preprint database ~~and both selling for 99c~~ with the first selling for 99c and the second free. Check them out! The former even lets you save papers for off-line reading.

For me at least, the hardest part of going through the arXiv emails every day was to pick out the interesting papers in the deluge of text. These apps do the right thing and segregate the categories and highlight the titles. Fitts’ Law in action — suddenly the daily ritual is orders of magnitude more pleasant!

Mar 9th, 2009| 05:18 pm | Posted by hlee

It bears the name of its inventor, Prasanta Chandra Mahalanobis. As opposed to the Euclidean distance, a household name, the name of this distance is rarely used but many pseudonyms exist with variations adapted into broad scientific disciplines and applications. Therefore, under different names, I believe that the Mahalanobis distance is frequently applied in exploring and analyzing astronomical data. Continue reading ‘[MADS] Mahalanobis distance’ »

Mar 6th, 2009| 03:42 pm | Posted by hlee

Ah ha~ Once I questioned, “what is systematic error?” (see [Q] systematic error.) Thanks to L. Lyons’ work discussed in [ArXiv] Particle Physics, I found this paper, titled **Systematic Errors** describing the concept and statistical inference related to **systematic errors** in the field of particle physics. It, gladly, shares lots of similarity with high energy astrophysics. Continue reading ‘systematic errors’ »

Tags:

coverage,

Heinrich,

likelihood,

Lyons,

nuisance parameter,

objective priors,

p-value,

particle physics,

statistical error,

subjective priors,

systematic error Category:

Algorithms,

arXiv,

Bayesian,

Cross-Cultural,

Data Processing,

Frequentist,

Jargon,

Misc,

News,

Physics,

Stat,

Uncertainty |

Comment
Mar 6th, 2009| 09:22 am | Posted by vlk

What XKCD says:

The mouseover text on the original says *“Correlation doesn’t imply causation, but it does waggle its eyebrows suggestively and gesture furtively while mouthing ‘look over there’.”*

It is a bad habit, hard to break, the temptation is great.