Organization of XRCF Calibration Data

Directory Structure

The data directory is currently on /hrc3.local/data/kobayash/data. This directory is accessible to anyone in the hrc user group. The directory structure is:

"DDD" represents the 3-digit day-of-the-year number. YYMMDD represents the standard year-month-day number. All dates and times are in UT/GMT.

HRC Telemetry

The HRC telemetry is saved in the raw telemetry format and compressed using gzip. Each file represents one hour of operation. The filename convention is DDDHHMMSS.tm.gz. (DDD=day of the year, HH=hour, MM=minute, SS=second, when data file was created.)

BND Data

There are six BND counters: four proportional counters at the HRMA, one at buliding 500, and a solid state detector at building 500. Each BND data set (run) is identified by a unique 6-digit Run ID. Typically one BND data set corresponds to one HRC test (i.e. one TRW ID), but occasionally it is restarted after a problem, resulting in multiple BND Run IDs that correspond to a single HRC test. Within each BND run, multiple spectra are taken with short integration times, typically order of 100 seconds.

There are two types of files. For each BND run there are 6 summary files (.sum files), one for each counter. (example) The summary file is a text file consisting of a long header followed by a single-line entry for each spectrum taken, listing the integration time, counts in ROI, etc.

The spectrum files (.pha) files contain the actual spectrum. (example) This is a simple text file, consisting of a header followed by the list of number of counts in each bin. In both files, the headers are prefixed by a "#" symbol and can be easily filtered out using grep, or other programs.

Filename convention for summary files is NNNNNNd?.sum, where NNNNNNN is the run ID and ? is the detector ID. For example, 113495d2.sum is the summary file for Run ID 113495, detector 2. Similarly, for spectrum files the convention is acqNNNNNNd?i*.pha, where '*' is the spectrum number. The first spectrum file corresponding to 113495d2.sum is therefore acq113493d2i0.pha. Note that there is no preceding zero for the spectrum number.

Unfortunately, there is no fixed rule to determine which detector corresponds to which detector ID number used in the filename. Detector 1 can be the North HRMA counter in one run and the solid state detector on the next. One needs to look in the header of the summary file to determine which counter the file corresponds to.

HRC EGSE files

The EGSE files (.rd files) are the same format we all know and love. The g-zipped rd file, the log file, and (in most cases) a g-zipped PostScript dump of the EGSE display are saved in the same directory. Steve's programs rd2ftm, ftm2prd, prd2epr, epr2img, etc., can be used to convert and/or analyse the data.

FAM data

The FAM data directory appears to be incomplete at this point. The files are saved in hour-long raw telemetry files and g-zipped. I do not know of any official software for reading these files.

Misc. Tools and Data

datasort2.current

This file currently resides in /hrc3.local/data/kobayash/shrc/analysis/miscdata/ directory. Each line in this file corresponds to a single EGSE file, and lists the corresponding TRW ID and BND Run ID, as well as important parameters such as x-ray energy, FAM and HRMA coordinates, X-ray source type, etc. It is a comma-separated plain text file. The fields are:

1. EGSE Filename
2. TRW ID Suffix
3. TRW ID
4. X-Ray Energy (keV)
5. HRMA Pitch (arcmin)
6. HRMA Yaw (arcmin)
7. FAM Y offset (mm)
8. FAM Z offset (mm)
9. FAM defocus (mm)
10. SIM Z offset (mm)
11. FAM Roll (arcmin)
12. FAM Pitch (arcmin)
13. FAM Yaw (arcmin)
14. BND directory and run ID (separated by colon)
15. Requested Run Time (sec)
16. Live Time (sec)
17. Total Event Count
18. X-ray Source Type

TRW ID Suffix is what is added to the TRW ID to distinguish multiple runs that correspond to a single CMDB entry, e.g. '-a', '-b'. The TRW ID recorded in entry 3 does include this suffix, but it is trivial to subtract entry 2 from entry 3 to get the 'bare' TRW ID (at least in shell scripts).

The same information is also availabe as an interactive HTML form.

CMDB

The CMDB file current as of the end of HRC testing is in /hrc3.local/data/kobayash/shrc/analysis/miscdata/ as old.run.cmdb. The TRW ID definitions and CMDB file format are available on-line.

Data analysis programs

What I believe to be the newest copies of Steve's data conversion/reduction programs are in /hrc3.local/data/kobayash/shrc/analysis/bin/ directory. IDL routines written at XRCF are in /hrc3.local/data/kobayash/shrc/analysis/idl_lib/. The IDL routines have not been catalogued and documented yet.

This directory can be added to the path by sourcing /hrc3.local/data/kobayash/shrc/analysis/hrc_env. This file also sets some other environmental variables such as IDL_PATH.

To get the data

A simple way to get an EGSE data is to use the script get_egse and get_egse_s. These scrips take the EGSE ID (filename minus extentions) as parameters. The scripts finds the .rd file in the appropriate directory, uncompress them and converts them to epr files. The resulting epr files are left in the current directory.