Comments on: [Q] Objectivity and Frequentist Statistics http://hea-www.harvard.edu/AstroStat/slog/2008/q-objectivity-frequentist/ Weaving together Astronomy+Statistics+Computer Science+Engineering+Intrumentation, far beyond the growing borders Fri, 01 Jun 2012 18:47:52 +0000 hourly 1 http://wordpress.org/?v=3.4 By: vlk http://hea-www.harvard.edu/AstroStat/slog/2008/q-objectivity-frequentist/comment-page-1/#comment-794 vlk Mon, 13 Oct 2008 20:30:58 +0000 http://hea-www.harvard.edu/AstroStat/slog/?p=901#comment-794 Ah. That's an interesting problem, and is perhaps related to the background estimation problem that Hyunsook is worrying about. But in this case, f1 is the same as f2, i.e., both detectors are observing the same thing. There is usually no problem if the detectors are perfectly calibrated with each other, but trouble arises when the statistical error is small in some sense compared to the difference between the two and there is no a priori reason to believe one detector over the other. Ah. That’s an interesting problem, and is perhaps related to the background estimation problem that Hyunsook is worrying about. But in this case, f1 is the same as f2, i.e., both detectors are observing the same thing. There is usually no problem if the detectors are perfectly calibrated with each other, but trouble arises when the statistical error is small in some sense compared to the difference between the two and there is no a priori reason to believe one detector over the other.

]]>
By: brianISU http://hea-www.harvard.edu/AstroStat/slog/2008/q-objectivity-frequentist/comment-page-1/#comment-792 brianISU Mon, 13 Oct 2008 18:20:57 +0000 http://hea-www.harvard.edu/AstroStat/slog/?p=901#comment-792 I forgot to add that one can have mle estimates of theta and plug these in to the mean of the finite mixture distribution. I forgot to add that one can have mle estimates of theta and plug these in to the mean of the finite mixture distribution.

]]>
By: brianISU http://hea-www.harvard.edu/AstroStat/slog/2008/q-objectivity-frequentist/comment-page-1/#comment-791 brianISU Mon, 13 Oct 2008 18:19:29 +0000 http://hea-www.harvard.edu/AstroStat/slog/?p=901#comment-791 Let me first comment on how I understand the problem just to make sure that my recommendation still fits and it agrees with your problem of interest. I understand the problem to be we have n1 observations from one detector and n2 observations from another. I am also assuming that these observations come from two independent distributions. So the n1 observations come from distribution f1 and similarly n2 observations come from distribution f2. Also, I am assuming that the interest is only on these two detectors and not the population of detectors (otherwise I would need a different model). So, with these two sets of observations, you would like an estimate of some mean given all the observations. Is this true? So, under the assumption I understand the problem, let N = n1 + n2. Also let pi1 = n1/N and pi2 = n2/N ( or some other weight scheme ). Then the total finite mixture distribution is: f(y|theta) = pi1*f1 + pi2*f2 (where theta is a parameter vector containing parameters from f1 and f2) and the expected values is just a weighted sum of expected values. Does this help? If not I can discuss a system reliability example. Let me first comment on how I understand the problem just to make sure that my recommendation still fits and it agrees with your problem of interest. I understand the problem to be we have n1 observations from one detector and n2 observations from another. I am also assuming that these observations come from two independent distributions. So the n1 observations come from distribution f1 and similarly n2 observations come from distribution f2. Also, I am assuming that the interest is only on these two detectors and not the population of detectors (otherwise I would need a different model). So, with these two sets of observations, you would like an estimate of some mean given all the observations. Is this true?

So, under the assumption I understand the problem, let N = n1 + n2. Also let pi1 = n1/N and pi2 = n2/N ( or some other weight scheme ). Then the total finite mixture distribution is:

f(y|theta) = pi1*f1 + pi2*f2 (where theta is a parameter vector containing parameters from f1 and f2)

and the expected values is just a weighted sum of expected values.
Does this help? If not I can discuss a system reliability example.

]]>
By: vlk http://hea-www.harvard.edu/AstroStat/slog/2008/q-objectivity-frequentist/comment-page-1/#comment-787 vlk Mon, 13 Oct 2008 17:14:40 +0000 http://hea-www.harvard.edu/AstroStat/slog/?p=901#comment-787 Hmm.. I am not sure I understand, could you give some sort of example? From what I understand, one would resort to a mixture model to explain multimodality in the posterior distribution, essentially assigning explanatory power for the different data to different components. A simple example that Hyunsook came up with to explain the probelm: suppose you ask a hundred men and ten women how bright the day was, and suppose the hundred men said it was 5+-0.3 (on a scale of 0-10), and the ten women said it was 7+-1.0 How bright <em>was</em> the day then? Hmm.. I am not sure I understand, could you give some sort of example? From what I understand, one would resort to a mixture model to explain multimodality in the posterior distribution, essentially assigning explanatory power for the different data to different components.

A simple example that Hyunsook came up with to explain the probelm: suppose you ask a hundred men and ten women how bright the day was, and suppose the hundred men said it was 5+-0.3 (on a scale of 0-10), and the ten women said it was 7+-1.0 How bright was the day then?

]]>
By: brianISU http://hea-www.harvard.edu/AstroStat/slog/2008/q-objectivity-frequentist/comment-page-1/#comment-786 brianISU Sat, 11 Oct 2008 16:39:23 +0000 http://hea-www.harvard.edu/AstroStat/slog/?p=901#comment-786 I am not sure if it was directly stated above, but one method is to use a finite mixture model. This problem even simplifies things since the proportion of observations coming from a specific distribution is known (meaning we won't need to estimate the proportions as well). Then, likelihood methods can be applied to estimate the parameters. With these estimates, any function of them will also be the maximum likelihood estimate and intervals estimates can also be obtained. I am not sure if it was directly stated above, but one method is to use a finite mixture model. This problem even simplifies things since the proportion of observations coming from a specific distribution is known (meaning we won’t need to estimate the proportions as well). Then, likelihood methods can be applied to estimate the parameters. With these estimates, any function of them will also be the maximum likelihood estimate and intervals estimates can also be obtained.

]]>
By: Paul Baines http://hea-www.harvard.edu/AstroStat/slog/2008/q-objectivity-frequentist/comment-page-1/#comment-784 Paul Baines Wed, 08 Oct 2008 04:43:38 +0000 http://hea-www.harvard.edu/AstroStat/slog/?p=901#comment-784 I don't know of any classic review paper, but: Sutton, A.J., and Higgins, J.P.T. (2007) Recent developments in meta-analysis, Statistics in Medicine, Vol. 27-5, pp.625-650 seems like a decent place to start. It has some basics, some history, and plenty of references for anyone interested. There is also a section in the red book (Bayesian Data Analysis by Gelman et al) that has an introduction and example. Meta-analysis is essentially hierarchical modelling though, the meta-analysis literature is just more concerned with the specific issues involved in combining different studies. Hope that helps... I don’t know of any classic review paper, but:
Sutton, A.J., and Higgins, J.P.T. (2007) Recent developments in meta-analysis, Statistics in Medicine, Vol. 27-5, pp.625-650
seems like a decent place to start. It has some basics, some history, and plenty of references for anyone interested. There is also a section in the red book (Bayesian Data Analysis by Gelman et al) that has an introduction and example. Meta-analysis is essentially hierarchical modelling though, the meta-analysis literature is just more concerned with the specific issues involved in combining different studies. Hope that helps…

]]>
By: vlk http://hea-www.harvard.edu/AstroStat/slog/2008/q-objectivity-frequentist/comment-page-1/#comment-783 vlk Tue, 07 Oct 2008 18:29:44 +0000 http://hea-www.harvard.edu/AstroStat/slog/?p=901#comment-783 Thanks, Paul, that's exactly the kind of situation here! Is there anything out there that is of relevance to (and comprehensible to) astronomers? Thanks, Paul, that’s exactly the kind of situation here! Is there anything out there that is of relevance to (and comprehensible to) astronomers?

]]>
By: Paul B http://hea-www.harvard.edu/AstroStat/slog/2008/q-objectivity-frequentist/comment-page-1/#comment-782 Paul B Tue, 07 Oct 2008 17:13:52 +0000 http://hea-www.harvard.edu/AstroStat/slog/?p=901#comment-782 Sorry I'm a bit slow in responding but I thought I'd point out that (if I understand your question correctly) there is a large amount of literature on this -- meta-analysis. Meta-analysis is the study of how to combine different studies of the same quantity, usually in the context of clinical/medical trials where different research groups have published different results with different sample sizes, error bars etc. Most of the advances in meta-analysis have come from the Bayesian perspective (as you may expect) but in short, if you want to combine them then you'll have to make some assumptions. It is an extremely active research field, especially with the proliferation of studies on the same topic -- combining them in some coherent manner is definitely challenging but potentially very powerful. Sorry I’m a bit slow in responding but I thought I’d point out that (if I understand your question correctly) there is a large amount of literature on this — meta-analysis. Meta-analysis is the study of how to combine different studies of the same quantity, usually in the context of clinical/medical trials where different research groups have published different results with different sample sizes, error bars etc. Most of the advances in meta-analysis have come from the Bayesian perspective (as you may expect) but in short, if you want to combine them then you’ll have to make some assumptions. It is an extremely active research field, especially with the proliferation of studies on the same topic — combining them in some coherent manner is definitely challenging but potentially very powerful.

]]>
By: vlk http://hea-www.harvard.edu/AstroStat/slog/2008/q-objectivity-frequentist/comment-page-1/#comment-780 vlk Fri, 03 Oct 2008 18:37:10 +0000 http://hea-www.harvard.edu/AstroStat/slog/?p=901#comment-780 Well, it could be because the first dataset oversamples the RMF, and the second integrates over the passband. Could be different telescopes entirely, with different exposure times, and with only a partial overlap in wavelength range. And also, as you say, I am concerned about possible systematic errors in one or both datasets which has nothing to do with the number of data points. Anyway, after considerable offline discussion, it came down to exactly that point. If you believe that there is some reason to believe one dataset over the other, find a weight that uses that information. Without that information, the only thing to do is lump them all together and pray. Well, it could be because the first dataset oversamples the RMF, and the second integrates over the passband. Could be different telescopes entirely, with different exposure times, and with only a partial overlap in wavelength range. And also, as you say, I am concerned about possible systematic errors in one or both datasets which has nothing to do with the number of data points.

Anyway, after considerable offline discussion, it came down to exactly that point. If you believe that there is some reason to believe one dataset over the other, find a weight that uses that information. Without that information, the only thing to do is lump them all together and pray.

]]>
By: Simon Vaughan http://hea-www.harvard.edu/AstroStat/slog/2008/q-objectivity-frequentist/comment-page-1/#comment-779 Simon Vaughan Fri, 03 Oct 2008 14:08:56 +0000 http://hea-www.harvard.edu/AstroStat/slog/?p=901#comment-779 On what basis are you concluding that the two spectra should have equal weight? If one dataset has 100 data points and S/N ~10 per datum, and the other has similar S/N but 10 data points, doesn't the first dataset contain more information (in either the Shannon or Fisher sense)? Is this because you are more concerned about a systematic error in one (or both) datasets, which is independent of the number of data points? On what basis are you concluding that the two spectra should have equal weight? If one dataset has 100 data points and S/N ~10 per datum, and the other has similar S/N but 10 data points, doesn’t the first dataset contain more information (in either the Shannon or Fisher sense)? Is this because you are more concerned about a systematic error in one (or both) datasets, which is independent of the number of data points?

]]>