Archive for the ‘Data Processing’ Category.

Sep 9th, 2011| 01:05 pm | Posted by vlk

We organized a Special Session on *Time Series in High Energy Astrophysics: Techniques Applicable to Multi-Dimensional Analysis* on Sep 7, 2011, at the AAS-HEAD conference at Newport, RI. The talks presented at the session are archived at http://hea-www.harvard.edu/AstroStat/#head2011

A tremendous amount of information is contained within the temporal variations of various measurable quantities, such as the energy distributions of the incident photons, the overall intensity of the source, and the spatial coherence of the variations. While the detection and interpretation of periodic variations is well studied, the same cannot be said for non-periodic behavior in a multi-dimensional domain. Methods to deal with such problems are still primitive, and any attempts at sophisticated analyses are carried out on a case-by-case basis. Some of the issues we seek to focus on are methods to deal with are:

* Stochastic variability

* Chaotic Quasi-periodic variability

* Irregular data gaps/unevenly sampled data

* Multi-dimensional analysis

* Transient classification

Our goal is to present some basic questions that require sophisticated temporal analysis in order for progress to be made. We plan to bring together astronomers and statisticians who are working in many different subfields so that an exchange of ideas can occur to motivate the development of sophisticated and generally applicable algorithms to astronomical time series data. We will review the problems and issues with current methodology from an algorithmic and statistical perspective and then look for improvements or for new methods and techniques.

Tags:

2011,

AAS,

HEAD,

September,

special session,

Timing Analysis Category:

Astro,

CHASC,

Data Processing,

Methods,

News,

Optical,

Stat,

Timing |

Comment
Nov 13th, 2009| 04:46 pm | Posted by hlee

I was told to stay away from python and I’ve obeyed the order sincerely. However, I collected the following stuffs several months back at the instance of hearing about import inference and I hate to see them getting obsolete. At that time, collecting these modules and getting through them could help me complete the first step toward the quest Learning Python (the first posting of this slog). Continue reading ‘some python modules’ »

Tags:

APLpy,

AstroPy,

IDLsave,

import inference,

libraries,

modules,

package,

Pyfits,

PyMC,

PyRAF,

PYSTAT,

Python,

PyWavelets Category:

Algorithms,

Astro,

Cross-Cultural,

Data Processing,

Jargon,

Languages,

Methods,

News,

Stat |

2 Comments
Oct 22nd, 2009| 07:08 pm | Posted by hlee

[arXiv:stat.ME:0910.2585]

Variable Selection and Updating In Model-Based Discriminant Analysis for High Dimensional Data with Food Authenticity Applications

by *Murphy, Dean, and Raftery*

Classifying or clustering (or semi supervised learning) spectra is a very challenging problem from collecting statistical-analysis-ready data to reducing the dimensionality without sacrificing complex information in each spectrum. Not only how to estimate spiky (not differentiable) curves via statistically well defined procedures of estimating equations but also how to transform data that match the regularity conditions in statistics is challenging.

Continue reading ‘[ArXiv] classifying spectra’ »

Tags:

BIC,

Classification,

clustering,

cross-validation,

curse of dimensionality,

discriminant analysis,

graphical model,

mclust,

model based,

semi-supervised learning,

statistical learning,

variable selection Category:

Algorithms,

arXiv,

Cross-Cultural,

Data Processing,

Jargon,

Methods,

Spectral,

Stat |

Comment
Oct 6th, 2009| 08:30 pm | Posted by hlee

Tags:

Classification,

clustering,

factor analysis,

Hubble,

multivariate analysis,

principle component analysis,

SING,

Spitzer,

tuning fork Category:

Algorithms,

Astro,

Cross-Cultural,

Data Processing,

Galaxies,

Jargon,

Methods,

Objects,

Stars,

Stat |

Comment
Oct 1st, 2009| 10:18 pm | Posted by hlee

I decide to discuss **Kalman Filter** a while ago for the slog after finding out that this popular methodology is rather underrepresented in astronomy. However, it is not completely missing from ADS. I see that the fulltext search and all bibliographic source search shows more results. Their use of **Kalman filter,** though, looked similar to the usage of “genetic algorithms” or “Bayes theorem.” Probably, the broad notion of **Kalman filter** makes it difficult my finding **Kalman Filter** applications by its name in astronomy since often wheels are reinvented (algorithms under different names have the same objective). Continue reading ‘[MADS] Kalman Filter’ »

Tags:

Cressie,

inference,

Kalman filter,

kriging,

MADS,

spatial statistics Category:

arXiv,

Astro,

Cross-Cultural,

Data Processing,

Imaging,

Jargon |

Comment
Oct 1st, 2009| 09:11 pm | Posted by hlee

So far, I didn’t complain much related to my “*statistician learning astronomy*” experience. Instead, I’ve been trying to emphasize how fascinating it is. I hope that more statisticians can join this adventure when statisticians’ insights are on demand more than ever. However, this positivity seems not working so far. In two years of this slog’s life, there’s no posting by a statistician, except one about BEHR. Statisticians are busy and well distracted by other fields with more tangible data sets. Or compared to other fields, too many obstacles and too high barriers exist in astronomy for statisticians to participate. I’d like to talk about these challenges from my ends.^{[1]} Continue reading ‘data analysis system and its documentation’ »

Tags:

ARF,

calibration,

ciao,

cultural shock,

data analysis system,

documentation,

FITS,

obstacles,

pha,

PSF,

RMF,

Sherpa,

standard procedure,

Tutorial,

unification,

validation,

XSPEC Category:

Astro,

Cross-Cultural,

Data Processing,

High-Energy,

Misc,

Quotes,

X-ray |

Comment
Sep 22nd, 2009| 12:03 pm | Posted by hlee

Thanks to a Korean solar physicist^{[1]} I was able to gather the following websites and some relevant information on Space Weather Forecast in action, not limited to literature nor toy data.

Continue reading ‘More on Space Weather’ »

Tags:

automatic,

CME,

computer vision,

data mining,

feature detection,

filament,

image processing,

machine learning,

manifold,

space weather,

statistical learning,

sunspot,

SVM Category:

Algorithms,

arXiv,

Cross-Cultural,

Data Processing,

Imaging,

Jargon |

Comment
Sep 10th, 2009| 11:20 pm | Posted by hlee

Soon it’ll not be qualified for [MADS] because I saw some abstracts with the phrase, **compressed sensing** from arxiv.org. Nonetheless, there’s one publication within refereed articles from ADS, so far.

http://adsabs.harvard.edu/abs/2009MNRAS.395.1733W.

*Title:***Compressed sensing imaging techniques for radio interferometry**

*Authors:* Wiaux, Y. et al. Continue reading ‘[MADS] compressed sensing’ »

Tags:

compressed sensing,

ill-posed,

image reconstruction,

interferometry,

inverse problem,

MADS,

Nyquist-Shannon sampling theorem Category:

Algorithms,

Cross-Cultural,

Data Processing,

Imaging,

Jargon,

Spectral |

Comment
Sep 8th, 2009| 10:17 am | Posted by hlee

I happened to observe a surge of principle component analysis (PCA) and independent component analysis (ICA) applications in astronomy. The PCA and ICA is used for separating mixed components with some assumptions. For the PCA, the decomposition happens by the assumption that original sources are orthogonal (uncorrelated) and mixed observations are approximated by multivariate normal distribution. For ICA, the assumptions is sources are independent and not gaussian (it grants one source component to be gaussian, though). Such assumptions allow to set dissimilarity measures and algorithms work toward maximize them. Continue reading ‘[ArXiv] component separation methods’ »

Sep 1st, 2009| 07:43 pm | Posted by hlee

[arxiv:0906.3662] **The Statistical Analysis of fMRI Data** by Martin A. Lindquist

Statistical Science, Vol. 23(4), pp. 439-464

This review paper offers some information and guidance of statistical image analysis for fMRI data that can be expanded to astronomical image data. I think that fMRI data contain similar challenges of astronomical images. As Lindquist said, collaboration helps to find shortcuts. I hope that introducing this paper helps further networking and collaboration between statisticians and astronomers.

**List of similarities** Continue reading ‘[ArXiv] Statistical Analysis of fMRI Data’ »

Tags:

data aquisition,

experimental design,

fMRI,

ICA,

image analysis,

image processing,

localization,

modeling,

pipeline,

preprocessing,

similarities,

Spatial,

temporal,

time series,

voxel Category:

arXiv,

Cross-Cultural,

Data Processing,

Imaging,

Jargon,

Methods,

Stat |

Comment
Aug 12th, 2009| 06:03 pm | Posted by hlee

Statistical Resampling Methods are rather unfamiliar among astronomers. Bootstrapping can be an exception but I felt like it’s still unrepresented. Seeing an recent review paper on **cross validation** from [arXiv] which describes basic notions in theoretical statistics, I couldn’t resist mentioning it here. **Cross validation** has been used in various statistical fields such as classification, density estimation, model selection, regression, to name a few. Continue reading ‘[ArXiv] Cross Validation’ »

Tags:

ADS,

cross-validation,

machine learning,

Model Selection,

n-fold Category:

arXiv,

Astro,

Bayesian,

Cross-Cultural,

Data Processing,

Frequentist,

Jargon,

Methods,

Quotes,

Stat |

Comment
Jul 29th, 2009| 01:02 am | Posted by hlee

Speaking of XAtlas from my previous post I tried another visualization tool called **Parallel Coordinates** on these Capella observations and two stars with multiple observations (AL Lac and IM Peg). As discussed in [MADS] Chernoff face, full description of the catalog is found from XAtlas website. The reason for choosing these stars is that among low mass stars, next to Capella (I showed 16), IM PEG (HD 21648, 8 times), and AR Lac (although different phases, 6 times) are most frequently observed. I was curious about which variation, within (statistical variation) and between (Capella, IM Peg, AL Lac), is dominant. How would they look like from the parametric space of High Resolution Grating Spectroscopy from Chandra? Continue reading ‘[MADS] Parallel Coordinates’ »

Tags:

Classification,

clustering,

display,

EDA,

eye catcher,

GGobi,

Inselberg,

parallel coordinates,

visualization Category:

Algorithms,

arXiv,

Cross-Cultural,

Data Processing,

High-Energy,

Jargon,

Methods,

Spectral,

X-ray |

3 Comments
Jul 12th, 2009| 07:21 pm | Posted by hlee

Approximately for a decade, there have been journals dedicated to **bioinformatics.** On the other hand, there is none in astronomy although astronomers have a long history of comprising a huge volume of catalogs and data archives. Prof. Bickel’s comment during his plenary lecture at the IMS-APRM particularly on **sparse matrix** and **philosophical issues on choosing principal components** led me to wonder why astronomers do not discuss **astroinformatics**. Continue reading ‘Astroinformatics’ »

Tags:

astroinformatics,

bioinformatics,

catalog,

dimension reduction,

journals,

penalize,

regularization,

sparse matrix,

variable selection Category:

Astro,

Cross-Cultural,

Data Processing,

Imaging,

Jargon,

Stat |

1 Comment
Jun 12th, 2009| 03:47 pm | Posted by hlee

**A Fast Thresholded Landweber Algorithm for Wavelet-Regularized Multidimensional Deconvolution**

Vonesch and Unser (2008)

IEEE Trans. Image Proc. vol. 17(4), pp. 539-549

Quoting the authors, I also like to say that __the recovery of the original image from the observed is an __**ill-posed** problem. They traced the efforts of wavelet regularization in deconvolution back to a few relatively recent publications by astronomers. Therefore, I guess the topic and algorithm of this paper could drag some attentions from astronomers. Continue reading ‘Wavelet-regularized image deconvolution’ »

Tags:

bound optimization,

deconvolution,

image processing,

impulse response,

MM algorithm,

PSF,

regularization,

restoration,

thresholding,

wavelet Category:

Algorithms,

arXiv,

Data Processing,

Fitting,

Imaging,

Jargon,

Methods,

Quotes,

Stat |

Comment
Jun 11th, 2009| 03:52 pm | Posted by hlee

I was at the SUSY 09 public lecture given by a Nobel laureate, Frank Wilczek of QCD (quantum chromodynamics). As far as I know SUSY is the abbreviation of **SUperSYmetricity** in particle physics. Finding such antimatter(? I’m afraid I read “Angels and Demons” too quickly) will explain the unification theory among electromagnetic, weak, and strong forces and even the gravitation according to the speaker’s graph. I’ll not go into the details of particle physics and the standard model. The reason is too obvious. Instead, I’d like to show this image from wikipedia and to discuss my related questions.

Continue reading ‘how to trace?’ »

Tags:

cliche,

collion,

identifiability,

identification,

irony,

LHC,

Power,

reconstruction,

source detection,

subparticle,

supersymmetry,

SUSY,

TRACE,

type I error,

Type II error,

uncertainty principle,

unification,

youtube Category:

Cross-Cultural,

Data Processing,

High-Energy,

Misc,

Quotes,

Uncertainty |

Comment