Comments on: On the unreliability of fitting http://hea-www.harvard.edu/AstroStat/slog/2007/on-the-unreliability-of-fitting/ Weaving together Astronomy+Statistics+Computer Science+Engineering+Intrumentation, far beyond the growing borders Fri, 01 Jun 2012 18:47:52 +0000 hourly 1 http://wordpress.org/?v=3.4 By: Jeff Scargle http://hea-www.harvard.edu/AstroStat/slog/2007/on-the-unreliability-of-fitting/comment-page-1/#comment-44 Jeff Scargle Tue, 10 Jul 2007 18:42:49 +0000 http://hea-www.harvard.edu/AstroStat/slog/2007/on-the-unreliability-of-fitting/#comment-44 ... so the challenge to machine learners is to develop procedures that can at the very least flag cases as "poor fit," and at the most respond to the situations that lead to poor fits in a robust way. Perhaps the latter would be in the form of a reported fit with large statistical and/or systematic errors reported. <blockquote>[<b>Response:</b> Partly that, yes. But often it is hard to know that a bad fit has been obtained until the error bars are determined, or the likelihood surface is computed. A particularly funny thing happened recently (or at least it will be funny in a year or so) when Brad W was trying to figure out the Ne/O abundance of a plasma -- he got one answer if he stepped through the values of Ne/O from low to high, and quite a different one if he went from high to low! There was a local minimum at one end which was trapping the solution, and because each successive fit started from where the previous one left off, it never got out of the rut! -vinay]</blockquote> … so the challenge to machine learners is to develop procedures that can at the very least flag cases as “poor fit,” and at the most respond to the situations that lead to poor fits in a robust way. Perhaps the latter would be in the form of a reported fit with large statistical and/or systematic errors reported.

[Response: Partly that, yes. But often it is hard to know that a bad fit has been obtained until the error bars are determined, or the likelihood surface is computed. A particularly funny thing happened recently (or at least it will be funny in a year or so) when Brad W was trying to figure out the Ne/O abundance of a plasma -- he got one answer if he stepped through the values of Ne/O from low to high, and quite a different one if he went from high to low! There was a local minimum at one end which was trapping the solution, and because each successive fit started from where the previous one left off, it never got out of the rut! -vinay]

]]>
By: The AstroStat Slog » All your bias are belong to us http://hea-www.harvard.edu/AstroStat/slog/2007/on-the-unreliability-of-fitting/comment-page-1/#comment-17 The AstroStat Slog » All your bias are belong to us Mon, 04 Jun 2007 21:42:48 +0000 http://hea-www.harvard.edu/AstroStat/slog/2007/on-the-unreliability-of-fitting/#comment-17 [...] School at Wallops Island (pdf1, pdf2), and no doubt part of the problem has to do with the (un)reliability of the fitting process when the chisq surface gets [...] [...] School at Wallops Island (pdf1, pdf2), and no doubt part of the problem has to do with the (un)reliability of the fitting process when the chisq surface gets [...]

]]>