This ChangeLog covers both the Funtools library and the suite of
applications. It will be updated as we continue to develop and improve
Funtools. The up-to-date version can be found
[The changelog for the initial development of Funtools, covering the
beta releases, can be found
Public Release 1.4.0 (15 August 2007)
- Public release of production-quality code, incorporating changes
and improvements from previous beta releases, including:
- Support for access to ASCII text column files.
- Support for fast indexed access of binary tables.
- Support for database views of tables, i.e. pre-set values for the filter
specification, the columns to activate, and display format.
- New programs include funcone (cone search), funindex (create index files),
and funcen (calculate centroids within regions).
Release 1.3.0b[n] (mainly internal SAO beta releases)
- Added -F[c] switch to change the column delimiter to the
- Extended fundisp's format switch (-f) so that it can now handle complex
formats such as 'x=sometext%3d- y=othertest%3d.ext'.
- Added support for creating and processing 1D FITS images.
- Added vcol=colname and vcol=/colname to filter specifications to
support use of a third value column when binning 2D images.
- Added switches to funcone to write out data rows are not within
any cone (-J, -X) and centers which have no close data rows (-L).
- In funjoin, added ability to specify a numeric tolerance for when joining
- shared memory support in gio now can create a shared segment if w+ is
specified as the open mode.
- Changed reggeometry man page so that examples correctly show angles
going counter-clockwise from the x-axis instead of from the y-axis.
- Added checks to funmerge to ensure that all files have the same columns.
- Fixed bug in text support that prevented header-less files from being
- Added support for 64-bit images (bitpix=64) and table columns (TFORM=K).
- Filter code was not applying bscale/bzero to columns.
- Fixed funimage bug that caused a .5/block error in WCS CRPIX values
generated from binary tables.
- Added feq(a,b) and div(a,b) macros to funcalc.
- Added support for single-line #define to funcalc.
- Updated wcs library to 3.6.6
- Fix bug in funcen in which ra,dec was not being calculated correctly
if physical and image coords did not match up.
- The filter syntax "col1 = col2" now explicitly generates an error
(you really want to do "col1 == col2").
- Added -o switch to include offset from the nominal target position.
- Fundisp now displays multi-dimensional vector columns properly.
- Documented support for lists of files processed as a single file
using "list: file1 ... filen" syntax.
- Fixed bugs in support for pipe file type (i.e. ability to pass
commands as a filename using "pipe: cmd arg1 ... argn" syntax).
- Fixed bug in funhist processing of image-based pixel histograms
(i.e using "xy" for columns) where a region was specified. All pixels
outside the region were erroneously being added to the bin containing
the 0 value.
- Disabled multi-file processing in funds9, which was breaking support
for pathnames containing spaces and is not used by ds9 anyway.
- Added support for Views of tables, i.e. pre-set values for the
filter specification, the columns to activate, and display format
(though the latter is for fundisp only).
- Added -l switch to funimage to read x, y, val columns from a list.
- Removed useless and meaningless section syntax foo'[*]' because it
breaks pointer de-referencing on string columns (i.e. foo'[*xxx=='a']').
Use foo'[*,*]' instead, as documented.
- String variables were not always being terminated properly in the
filter code because FITS 'A' data is not necessarily null-terminated.
- Added funtools version number to all usage() displays.
- Added explanation of switch arguments to many usage() displays.
- The filter keyword row# now supports single row selection as well
as range selection, i.e., "row#=100" along with previous "row#=100:200".
- fundisp now outputs "0x" before hex values.
- Fixed bug in filter parser which processed rangelists incorrectly
if spaces were put into the rangelist (i.e. "pha= 1 : 3" instead of
- Fixed a bug in funindex which created a wrongly named index file
if more than one "." was in the input file name.
- Added support to funcone to take ra, dec, radius from a list
(i.e. columns in a FITS file or a text file).
- Fixed a bug in FunColumnActivate so that if some columns are
explicitly activated while others are de-activated, only the
explicitly activated columns are activated (code was activating all
columns in this case).
- Fixed a bug in funindex which prevented indexing tables containing
a column named N.
- fundisp now encloses ASCII column values in single quotes (unless
-T is specified to output RDB format).
- If a filter specification only involves indexed columns, then the
compiled filter is not used.
- Funmerge can now be given a list of files to merge using @list syntax.
Also removed the restriction on how many files can be merged (was limited to
the max number of open files).
- Added ability to edit (add, delete, modify) header parameters in funhead
by specifying an output file (editing acts as a filter) and an edit command
file (which can be stdin).
- Funtools now contains preliminary code to support (fast) indexed access
of binary tables. See idx.html or "man funidx" for more details.
- Funtools now contains preliminary code supporting access to ASCII
column files. See text.html or "man funtext" for more details.
- Fixed bug in funcalc in which columns used in an expression were
always being replaced by new columns, with all associated parameters
(e.g. WCS) were being deleted. Now this only happens if the column
explicitly changes its data type.
- Fixed bug in funcalc in which the raw data and user data became out
of sync for one row after every 8192 (FUN_MAXROW) rows.
- Fixed bug in gio in which gseek returned 0 instead of the current byte
offset for disk files.
- Added funcone program to perform cone search on RA, Dec columns in
a FITS binary table.
- Fixed bug in polygon, pie and rotated box region filtering for
tables (nearby rows exactly in line between two non-vertical or
non-horizontal vertices were being accepted incorrectly).
- Fixed pie and panda regions so that the angles now start from
positive x axis == 0 degrees and run counter-clockwise, as
documented. They were going from positive y. NB: a similar change
was made to ds9 release 4.0b3. You must be using ds9 4.0b3 or later
in order to have the correct behavior when generating regions in ds9
and using them in funtools.
- Added -p [prog] switch to funcalc to save the generated program.
instead of executing (and deleting) it.
- Upgraded zlib to 1.2.3.
Patch Release 1.2.4 (internal SAO and beta release only)
- In funcalc, added support for user-specified arguments via the
-a [argstr] switch. These arguments are accessed in the compiled program
using the supplied ARGC and ARGV(n) macros.
- Added -n (no header display) to fundisp to skip outputting header.
- Added checks for various types of blank filters.
- Added macros NROW (current row number) and WRITE_ROW (write current
row to disk) to funcalc.
- funcalc no longer requires that at least one data column be
specified in the compiled expression.
- Added FUN_NROWS to FunInfoGet() to return the total number of rows in
an input table (i.e. value of NAXIS2).
- The compiled funcalc program now includes stdlib.h and unistd.h.
- The util/NaN.h header file is now modified at configure time to
contain endian status for the target architecture. References to
specific platforms have been removed.
- Added -m switch to funtable to output multiple files, one for
each input region (and a separate file for events that pass the
filters but are not in any region).
- Added ability to add new parameters (FunParamPutx) after writing
data if space is previously reserved in the form of a blank parameter
whose value is the name of the param to be updated. (Also requires the
append argument of FunParamPutx be set to 2).
- Added ability to build shared libraries. With --enable-shared=yes,
shared library is built but not used. With --enable-shared=link,
shared library is linked against (requires proper installation and/or
use of LD_LIBRARY_PATH).
- Added -v [column] support to funcnts so that counts in a table
can be accumulated using values from a specified column (instead of
the default case where an integral count is accumulated for each event
in a region).
- Added funcen program to calculate centroids within regions
(binary tables only). Also added support for a funcen-based centroid
tool to funtools.ds9.
- Fixed bug which prevented successful filtering of columns containing
- Added filter check to ensure that a column is not incorrectly used
as an array.
- Fundisp now displays column arrays indexed from 0, not 1.
- Added -i [interval] support to funcnts so that multiple intervals
can be processed in a single pass through the data. For example,
specifying -i "pha=1:5;pha=6:10;pha=11:15" will generate results in
each of 3 pha bands.
- Fixed calculation of LTV quantities when binning floating point
column data (value was off by 0.5).
- Added support for 'D' in floating point header values.
- Added -a switch to funimage and funtable to append output image or
table to an existing FITS file (as an IMAGE or BINTABLE extension).
- Added support for column scaling (TSCAL and TZERO) on input columns.
Note that the default column type is changed to accommodate scaling (e.g.
a column of type 'I' is changed to 'J', 'J' is changed to 'D') so that
the scaled values can be handled properly by programs such as fundisp
(which utilize default types).
- Added support to FunColumnSelect() for handling structs of arrays
(i.e. where returned columns are contiguous) instead of the default array
of structs (returned row are contiguous). This is done by specifying
"org=structofarrays" in the plist and passing a single struct containing
- When writing an rdb/starbase file, fundisp now outputs the full
column name, regardless of the width of the column (which ordinarily
is truncated to match).
- Fixed support for large files by changing all file positions variables
from "long" declarations to "off_t.
- Fixed bug in funcalc incorrectly processed multiple array
references (e.g. cur->foo=cur->x;cur->foo=cur->y;) within a single
line of code.
- Added FILTER_CFLAGS environment variable for all filtering. Also added
--with-filter-cc and --with-filter-cflags options on configure to allow
specification of a default C compiler and associated CFLAGS for filtering.
All of this is necessary in order to support 64-bit libraries under Solaris.
- Added the funtbl script to extract a table from Funtools ASCII output.
- Added code to funimage to update IRAF DATASEC keyword.
- Added checks to ensure that image dimensions are positive.
- Fixed a bug in funimage where int data was being scaled using BSCALE and
BZERO but these keywords also were being retained in the output image header.
Now the data are not scaled unless the output data type is float (in which
case the scaling parameters are removed).
- Fixed a bug in funmerge which prevented merging of files unless one of
the -f, -w, or -x switches were used.
- Fixed a bug in funtable and fundisp which caused the special '$n' column
to be output incorrectly.
- Fixed sort option in funtable, which previously worked only if the
record size was an even divisor of 8192 (and returned garbage otherwise).
- Fixed bug in filters involving FITS data type 'X' (bitfield).
- Fixed bug in funcnts in which the output angles and radii were
being displayed incorrectly when multiple panda shapes were specified.
- Fixed bug in pandas and pies using n= syntax when first angle
specified was greater than second. The resulting mask was of
the correct shape but contained only a single region.
- Table row access routines will now decrease maxrows if memory cannot be
allocated for maxrows*sizeof(row), i.e. if the size of a row is so large that
space for maxrows cannot be allocated.
- The FUN_MAXBUFSIZE environment variable was added to limit the
max buffer size that will be allocated to hold table row data. The
default is 5Mb.
- Generated PostScript and PDF versions of the help pages.
- Moved OPTIONS section before (often-lengthy) DESCRIPTION section in
- All memory allocation now does error checking on the result
(except wcs library, which is external code).
- Removed some compiler warnings that surfaced when using gcc -O2.
- Updated wcs library to 3.5.5.
- Upgraded zlib to 1.2.1.
Patch Release 1.2.3 (12 January 2004)
- Generated man pages from the html pages. These are installed
automatically at build time.
- Changed instances of sprintf() to snprintf() to protect
against buffer overflow.
- Fixed a number of compiler warnings in non-ANSI compilers.
- Increased SZ_LINE parameter value from 1024 to 4096.
Patch Release 1.2.3b1 (19 August 2003)
- The rule for using comma to separate a table filter expression
and a region expression has been changed. The rule now states:
This rule handles the cases of pure regions and pure column filters properly.
It unambiguously assigns the boolean and to all mixed cases. Thus:
- if both expressions contain a region, the operator used is or.
- if one (or both) expression(s) does not contain a region, the operator
used is and.
both are equivalent to:
foo.fits[circle(10,10,3) && pi=1:5]
- When include files are used in filters, they now have implied
parentheses surrounding them. Thus, if a region file foo.reg contains two
regions (e.g. circle 1 2 3 and circle 4 5 6), the syntax:
is equivalent to:
pha=4:5 && (circle 1 2 3 || cir 4 5 6)
pha=4:5 && circle 1 2 3 || cir 4 5 6
and the pha filter is applied to both regions.
- Filters and comments now can be terminated with the string
literal "\n" as well as ";" and the new-line character. This means
that a region can have comments embedded in it:
funcnts foo.fits "circle 512 512 10 # color=red\n circle 512 512 20"
- Added capability to update the value of an existing parameter
after writing the table or image (assuming the output image is a
disk file or is being redirected into a file).
- Improved handling of parentheses in filter expressions.
- Fixed a bug in image (not event) regions in which circles and
annuli with radius of 1 pixel were not being processed. No counts and
no area would be found in such regions.
- Fixed a bug in funcnts in which the radii column values for out of sync
if multiple annuli were specified (instead of a single varargs or accel
- By default, fundisp will display integer image data as floats
if the BSCALE and BZERO header parameters are present.
- Added -L switch to funhead to output starbase list format.
- Changed the name of the routine _FunColumnSelect to
FunColumnSelectArr, in order to emphasize that it is not
a private routine.
- Funcalc now checks to ensure that a column was specified as part of
- Funcalc local variables in the compiled program now use a "__" prefix
to avoid conflicts with user-defined variables.
- Unofficial unsigned short (bitpix=-16) image data now is scaled
correctly using BSCALE and BZERO header parameters.
- Ported to Intel icc and gcc 3.3 compilers.
- Updated wcs library to 3.5.1.
- Changed license from public domain to GNU GPL.
Patch Release 1.2.2 (18 May 2003)
- Fixed funcalc so that it now actually compiles an expression and
runs it, instead of getting a "filter compilation error". Oops!
- Fixed bug in FunOpen in which the bracket specification was being
removed from the filename if a disk file was opened for "w" or "a".
- Fixed bug in FunFlush which prevented two successive calls to
FunImagePut from writing the second extension header properly.
- All filter routines now use gerror(stderr, ...) call instead of
fprintf(stderr, ...) so that output to stderr can be turned off (via
setgerror(level) or GERROR environment variable).
- All standard Funtools programs check for GERROR environment
variable before setting gerror flag.
- Some error messages about invalid region arguments were not being
- FITS parameters/headers now conform more closely to FITS standard:
- Blank keywords are treated in the same way as COMMENTS and HISTORY cards
- XTENSION keywords are now exactly 8 characters long
- 'E' is output instead of 'e' in floating point param values
- PCOUNT and GCOUNT are output correctly for image extensions
- EXTEND=T is output in primary header
- COMMENTS and HISTORY start in column 9
Patch Release 1.2.1 (24 April 2003)
- Varargs ellipse and box annular regions were being
processed incorrectly when the following conditions all were met:
Varargs ellipse and boxes are regions of the form:
- the region was specified in physical or wcs coordinates
- the data file contained LTM/LTV keywords, i.e., it
was blocked with respect to the original data file
- the program being run was an image program (e.g. funcnts, funimage)
ellipse x y a1 b1 a2 b2 ... an bn [angle]
box x y l1 w1 l2 w2 ... ln wn [angle]
where at least 2 sets of axis (length) values were specified to form
an annulus (i.e. simple ellipses and boxes worked properly). With all
of the above conditions met, a region in physical coordinates saw its
second length argument converted incorrectly from physical coordinates
to image coordinates. In simple terms, this means that funcnts did not
process elliptical or box regions in physical coords on blocked images
properly. Note that blocking on the command line (e.g. foo.fits[*,*,2])
did work when no LTM/LTV keywords existed in the file.
- The fundisp -f switch now supports specification of
column-specific display formats as well as a more convenient way to
specify datatype-specific display formats. Both use keyword=value
specifiers. For columns, use:
fundisp -f "colname1=format1 colname2=format2 ..." ...
fundisp -f "time=%13.2f pha=%3d" ...
You also can specify display formats for individual datatypes using the FITS
binary table TFORM variables as the keywords:
fundisp -f "D=double_format E=float_format J=int_format etc."
fundisp -f "D=%13.2f I=%3d" ...
The old position-dependent syntax is deprecated.
- Fundisp will now print out a single 16-bit (or 32-bit) unsigned
int for a column whose data format is 16X (or 32X), instead of
printing 2 (or 4) unsigned chars.
- Fixed bug in which fundisp was not able to display bitfield data for
raw event lists.
- Previously, when binning columns used implicitly in a region
and explicitly in a filter could suffer from a case sensitivity problem.
This has been fixed.
- Fixed internal mask=all switch on fundisp.
- Filter include files now simply include text without changing the state
of the filter. They therefore can be used in expression. That is, if foo1
contains "pi==1" and foo2 contains "pha==2" then the following expressions
"[@foo1&&@foo2]" is equivalent to "[pi==1&&pha==2]"
"[pha==1||@foo2]" is equivalent to "[pi==1||pha==2]"
"[@foo1,@foo2]" is equivalent to "[pi==1,pha==2]"
- Fixed bug in filter specification which caused a SEGV if a varargs-style
region was enclosed in parens.
- Updated wcs library to 3.3.2.
Public Release 1.2.0 (24 March 2003)
- BSCALE and BZERO are now always applied to int pixel data, instead of
only being applied if the desired output is floating point.
Beta Release 1.2.b3 (4 February 2003)
- In FunColumnSelect, added the ability to specify an offset into
an array in the type specification, using the extended syntax:
The [poff] string specifies the offset. For example, a type specification
such as "@I" specifies the third (i.e., starting from 0) element in
the array pointed to by the pointer value. A value of "@2I" specifies
the fifth and sixth values in the array.
- Added a non-varargs version of FunColumnSelect called _FunColumnSelect:
int _FunColumnSelect(Fun fun, int size, char *plist,
char **names, char **types, char **modes, int *offsets,
- Added support for sorting binary tables by column name using:
funtable -s "col1 col2 ... coln" ...
- Added the FUN_RAW macro which, when applied to the "name" parameter
of FunParamGets(), returns the 80-character raw FITS card instead of
only the value.
- Added support for comparing column values with binary masks of the
form 0b+, e.g.:
Previously, such masks had to be specified in decimal, octal, or hex.
- Completed support for type 'L' (logical) in fundisp and in filtering of
- Fixed bug in funhist that was improperly setting the number of bins
when the data was of type float.
- Fixed bug in filter/Makefile where the filter OBJPATH #define was
being passed to the wrong module.
Beta Release 1.2.b2 (7 October 2002)
- Updated wcs library to 3.1.3.
- Added support for reading gzip'ed files via stdin.
Beta Release 1.2.b1 (24 September 2002)
- Added the following accelerators to region filtering:
BOX xcenter ycenter xw1 yh1 xw2 yh2 ... xwn yhn (angle)
BOX xcenter ycenter xwlo yhin xwout yhhi n=[number] (angle)
CIRCLE xcenter ycenter r1 r2 ... rn # same as annulus
CIRCLE xcenter ycenter rinner router n=[number] # same as annulus
ELLIPSE xcenter ycenter xw1 yh1 xw2 yh2 ... xwn yhn (angle)
ELLIPSE xcenter ycenter xwlo yhin xwout yhhi n=[number] (angle)
- Added the following new pandas (Pie AND Annulus) to region filtering:
CPANDA xcen ycen ang1 ang2 nang irad orad nrad # same as panda
BPANDA xcen ycen ang1 ang2 nang ixlo iylo ixhi iyhi nrad (ang) # box
EPANDA xcen ycen ang1 ang2 nang ixlo iylo ixhi iyhi nrad (ang) # ellipse
- Added support for filtering images using simple FITS image masks,
i.e. 8-bit or 16-bit FITS images where the value of a pixel is the
region id number for that pixel (and therefore must be greater than
0). The image section being filtered must either be the same size as the
mask dimensions or else be an even multiple of the mask. This works with
image-style filtering, i.e., funcnts can utilize a mask on both
images and binary tables.
- Added '$n' to fundisp column specification to allow display of
ordinal value of each row passing the filter.
- Added code to support region filtering on image sections.
- Fixed bugs which prevented filtering more than one ASCII region file.
- Fixed bug occasionally causing filter slave processes to become zombies.
- Fixed bugs in event filtering: annulus with inner radius of 0
(i.e., a circle) was rejecting events with coordinates xcen, ycen.
Also, pie with angles of 0 and 360 was rejecting some events.
Image filtering (e.g. funcnts) did not have these problems.
- Filters now accept global exclude regions without an include region.
In such a case, the field region is implied. That is, "-circle(x,y,r)"
is equivalent to "field; -circle(x,y,r)", etc.
- Fixed panda so that it can be used as a global exclude.
- Allow empty ds9 region file (comments and globals only) to be
a valid filter. Totally ignore zero length region or include file.
- Fixed funcnts bug that was displaying 0 value as inner radius of
a circle, instead of just one radius value.
Public Release 1.1.0 (22 April 2002)
New features include:
- Funtools programs now accept gzip'ed files as valid input.
- Improved security via replacement of system() function.
- fundisp, funcnts, funhist can output starbase/rdb format (tabs between columns, form-feeds between tables).
- Improved support for Windows platform, as well as new support for Mac OSX.
Pre-Release 1.1.0e (10 April 2002)
- Added enough support to skip over variable length arrays in BINTABLES.
We will add full support if this non-standard construct becomes more widely
- Fixed bug in underlying fitsy _gread() routine that was returning
an arbitrary bytes-read value if the input fd was invalid.
Pre-Release 1.1.0e (19 March 2002)
- Added additional check for Windows/PC to filter/Nan.h.
- Upgraded zlib library to 1.1.4 (fix double free security hole).
Pre-Release 1.1.0e (27 February 2002)
- Changed filter/process.[ch] to filter/zprocess.[ch] to avoid name
collision with Cygwin include file.
- Added -a switch to funhead to display all headers in a FITS file.
Pre-Release 1.1.0e (11 February 2002)
- Fixed filter parser so that it ignores ds9 "ruler" and "text" markers
only up to the first \n or ; (was ignoring to last \n).
- The NBLOCK parameter in fitsy/headdata.c was too large for Mac OS X
(max size of a declared char buf seems to be about .5 Mb).
Beta Release 1.0.1b5 (31 January 2002)
- Fixed bug introduced in calculated IRAF LTM values in 1.0.1b3.
- Fixed bug in filter parser giving wrong answers when two range
lists were combined with and explicit boolean operator:
$ fundisp $S"[x=512&&y=511,512]"
incorrectly acted like:
In general, we recommend use of explicit parentheses.
- Fixed filter/NaN.h to recognize Compaq Alpha again (broken by their last change to cc).
- Removed redundant varargs definitions that conflicted with Alpha compiler definitions.
- Added blank line to inc.sed to work around Apple Mac OS X bug in which the
"i" (insert) command was treating final \\ as continuation \ in the text.
- Added include of mkrtemp.h to mkrtemp.c to get conditional compilation
for Mac OSX.
- Added support for --with-zlib to fitsy so that ds9 could use its own
copy of zlib (and not build the copy in fitsy).
- Removed config.cache and Makefile files from distribution tar file.
Beta Release 1.0.1b4 (26 January 2002)
- Make explicit that column filters are not permitted in an image
expression (such as the funcnts region arguments).
- Fix bug in region parser in which a region (without parens),
followed immediately by an operator:
circle 512 512 .5&&pi==1
was not processing the final argument of the region correctly.
- Ignore new "tile" directive in filters (used by ds9).
Beta Release 1.0.1b3 (4 January 2002)
- Made modifications to Makefile.in to make releases easier.
- Added instructions Makefile.in so that funtools.h will always
have correct #defines for FUN_VERSION, FUN_MAJOR_VERSION,
FUN_MINOR_VERSION, and FUN_PATCH_LEVEL.
- Allow #include statements in funcalc program files.
- funimage now updates all 4 CDX_Y values by the block factor.
- Minor changes to make funtools work under darwin (Mac OS X).
Beta Release 1.0.1b2 (14 November 2001)
- Fixed FunOpen() bug (introduced in b1) in which filenames without
extensions SEGV'ed on open. Yikes!
- Funmerge now extends the tlmin/tlmax values of the output
binning columns so that merged events from widely separated files are
valid in the output table.
- In funhist, added -w switch to specify bin width (lo:hi:width)
instead of number of bins (lo:hi:num). Added support for this new
width option in funtools.ds9.
- If a tdbin value was set using bincols=(name:tlmin:tlmax:tdbin, ...),
the WCS parameters were not being updated properly.
- Cleaned up build support for zlib.
Beta Release 1.0.1b1 (6 November 2001)
Public Release 1.0.0 (31 July 2001)
- "a new day with no mistakes ... yet"
Index to the Funtools Help Pages
Last updated: 22 April 2002