Simulations are an easy way to proceed when mathematical calculations are difficult. Simulate 100 data sets WITH the supposed feature and a given exposure time. Run a statistical analysis of each and see what percent of the time you get a positive detection. This is (as Simon says) a power calculation. I’m surprised it is not standard. This NIH (US National Institute of Health, the main US Health Studies Funding Agency) would never fund something without a power calculation!

With only one simulation you could be very mislead. If that were not the case, the exposure time is probably too long. You are interested in using just enough exposure time to see the feature (to save/share resources, I assume). Thus the power should not be 100%.

Perhaps you meant to give examples of how one goes about doing a power calculation. That will be useful, certainly. But in that case, all you need to say in your prescription is “generate dataset from model here” (“generate_fake_dataset()”) and it will be understood what is meant.

2. If vlk’s comment is an answer to to implement the power calculation, I’m disappointed because the comment sounds like one is selling a drug before the completion of a clinical trial. Consider a prescription drug known for curing a disease. Unfortunately, there were a few reports (or some doubts) that the drug could have killed/deteriorated patients. How could we tell the drug actually killed/could kill a person? Since FAKEIT simulates spectra according to exposure time, we could build an empirical power function along exposure time. In stead of quitting the clinical trials, it would be nice to attempt a study with some simple source models by utilizing the ready implemented modules (packages).

I guess Simon Vaughan has the sketch to resolve the problem. Can it be shared???

But then the bureaucrats would worry the oversubscription rate was dropping! (You cannot please everyone all the time.)

A word of warning also, that proposals should not be held to the same standard as manuscripts submitted to journals (because proposals are usually written in a tearing hurry, and the space limitations can be a crippling limitation, and it is simply not fair to expect the same analysis effort spent on simulated data as would be on real data), so some sloppiness is acceptable.

It would indeed be interesting to compare the results of feasibility calculations to those from the real data, but I don’t see how this is possible since proposals are always kept secret. Maybe panels should be asked to log a one line description of the ‘significance’ (or similar) that is claimed in the proposals, which could later be compared to the result from the real data by the mission teams. They could then compare the statistics without revealing any private information on specific proposals. (Trouble is, no panel or mission team wants the extra work, and publication bias means that the ‘significance’ of the intended result may never be published if it is not as high as advertised in the proposal!)

Again, medical trials are ahead of us (in this respect). It is now standard (at least in the UK) to log the details of a medical trial (science goal, sample size, duration, etc) before it is performed. That way there’s a record of every trial, whether or not the results are published. The number of non-results can then be estimated. I should point out that what concerns me is the lack of a result, not an interesting null result. If a predicted feature is not confirmed, at high ‘significance’, that is all well and good, and potentially interesting. But if the data are inadequate to tell either way then the proposal was a waste of telescope time.

But what about unexpected results, discoveries that are hard to predict? The observations that bring exactly what we have planned for in the simulations are very good, but may not be as exciting as the ones that carry the discoveries. How many of those have we had recently? Plenty of X-ray jets, unpredicted spectral features, interesting morphology of X-ray gas etc…. How can the panel foresee those in the final decision?