Note: CalFUSE v3.1 employs a new scheme to encode the data quality of the jitter files. As a result, old jitter files will not work properly with the new pipeline. It's easy to generate new ones; just follow the instructions below.
Introduction
CalFUSE v3.1 is the latest version of the FUSE data-reduction pipeline. The big change from previous versions is that we now maintain the data as a photon list (called an intermediate data file, or IDF) throughout the pipeline. Bad photons are flagged but not discarded, so the user can examine, filter, and combine data without re-running the pipeline.
A number of design changes enable the new pipeline to run faster and use less disk space than before: We've combined 24 modules into 8. Data are read once at the start of each module and written once at the end. All changes are made to the IDF; no additional files are created. Finally, we use a new format for the IDF file that increases i/o speed by an order of magnitude.
Two obvious differences in the output spectral files produced by CalFUSE are that spectra are extracted only for the target aperture (non-target extractions are possible, but require human intervention), and that the spectra are binned in wavelength. The binning can be set by the user, but the default is 0.013 Å.
Though CalFUSE v3.1 incorporates numerous upgrades and bug fixes relative to v3.0, the code is not perfect. Please look carefully at your output and report any oddities or suggestions for improvement to fuse_support@pha.jhu.edu.
Recent Improvements
New in CalFUSE v3.1.0
Jitter Files: In v3.0, data obtained during periods of uncertain pointing information were often flagged as bad, even though the data were fine. In v3.1, jitter files are generated using a simpler, more robust algorithm to determine when pointing information is reliable. In the absence of good pointing information, the pipeline does nothing, rather than flagging the data as bad. As a result, the meaning of the jitter-file TRKFLG array has changed: it's now 5 when tracking is good, 3 when tracking is bad, and 0 when tracking is unknown. If you use old jitter files with the new software, the pipeline may reject good data or keep bad data. If you have the housekeeping files, you can generate jitter files that are consistent with the new pipeline. To do it, delete (or hide) the old jitter files, then type
prompt> cf_jitter P9990101001hskpf.fit P9990101001jitrf.fit
for each exposure. Run the pipeline as usual.
High Voltage: In v3.0, data obtained when the detector voltage was less than 95% of the expected value were flagged as bad. Now, for detectors 1A, 1B, and 2B, the level is 85%. For detector 2A, data with HV > 90% is processed without comment, data with HV > 80% generates a warning message, and data with HV < 80% is flagged as bad. For detector 2A data in the 80-90% range, the user should examine the distribution of pulse heights in the target aperture to ensure that the peak of the distribution is well separated from the background.
Extraction Windows: In v3.0, the geometric-distortion correction moved small segments of the HIST spectra out of their extraction windows. The solution was to increase the size of the HIST extraction windows in Y. A new keyword, HIST_PAD, which varies with both the detector and the aperture, is read from the header of the corresponding extension of the CHID_CAL file. While making this change, we found and corrected a bug in the calculation of the spectral Y centroid.
Walk Correction: In v3.0, the HIST and TTAG wavelength scales did not agree. This was a result of the walk correction, which was applied to TTAG but not HIST data. Since the walk correction is never zero, HIST data were never fully converted to the FARF. We now assign a pulse height (the mean for a particular date and aperture) to each HIST photon and apply the walk correction to all data. While this trick reduces the large-scale errors in the HIST wavelength scale, users should remember the that small-scale errors due to localized regions of gain sag cannot be corrected in HIST data.
Background Subtraction: When scaling the model background image to match the data, the subroutine cf_scale_bkgd (called cf_ttag_bkgd in v3.0) did not properly account for regions of the detector excluded because of airglow contamination. As a result, it underestimated the total background. The program has been extensively revised.
Spectral File Format: The final extracted spectral (*fcal.fit) files now have the standard FITS binary table format that was employed in old (v2.4 and earlier) versions of the pipeline. The content of these files (WAVE, FLUX, ERROR, COUNTS, WEIGHTS, BKGD, and QUALITY arrays) is the same as in v3.0.
IDF File Format: Errors in the calculation of TIME_SUNRISE and TIME_SUNSET have been corrected. As these values are never greater than 6000 s, they are now stored (and treated) as shorts.
Stim Pulses: If the pipeline cannot compute the centroids of the detector stim pulses, it reads a new calibration file (STIM_CAL), in which the time-dependent average values of the drift coefficients are tabulated.
Background Count Rate: The burst-detection algorithm computes the count rate on the unilluminated regions of the detector and writes it to the IDF. It now does so for the entire exposure, not just for times previously identified as good.
New Header Keywords: Several keywords have been added to the IDF file headers, including the limb-angle limits BRITLIMB and DARKLIMB from the SCRN_CAL file and the calculated orbital period ORBPERID.
Airglow Calibration Files: The AIRG_CAL files, which identify regions of each detector likely to be contaminated by airglow features, are now expressed in FARF, rather than RAW, coordinates. This change allows us to use smaller airglow windows and thus to discard less data.
Pulse-Height Limits: The default pulse-height limits (both upper and lower) have been modified for all four detectors.
New in CalFUSE v3.1.1
Locating the Spectra: We use a new algorithm to find spectra on the detector. First, the position of the airglow spectrum in the LWRS aperture is used to constrain the position of the airglow spectra of the smaller apertures. Then, the airglow spectrum of the target aperture is used to constrain the position of the target spectrum.
New in CalFUSE v3.1.2
Extraction Windows: In v3.0, a bug in cf_extract_spectra caused it always to use the smaller, point-source extraction window, even for extended sources. This could lead to an underestimate of the flux for extended sources. The program now uses either the point-source or extended-source aperture, as appropriate.
New in CalFUSE v3.1.3
FPA Motions: We have long corrected the X positions of photon events for X offsets of the focal-plane assemblies (FPA's), which contain the spectrograph apertures. It turns out that moving the FPA's in the Z direction also shifts the target spectra in X. This hasn't mattered before, but the recent decision to move the LiF2 FPA away from the spectrograph focal plane makes this correction important, and it is included in v3.1.3.
Overview of the Pipeline
The following modules are called by the shell script calfuse.csh:
cf_ttag_init, cf_hist_init: Converts raw-data file into an IDF containing photon-event list, GTI's, and timeline table. Histogram data are converted to a pseudo-time-tag format.
cf_convert_to_farf: Corrects for detector deadtime. Transforms photon coordinates into the FARF (flight alignment reference frame).
cf_remove_motions: Corrects for mirror, grating, FPA, and spacecraft motions.
cf_assign_wavelength: Applies astigmatism and Doppler corrections and assigns a wavelength to each photon.
cf_screen_photons: Checks data quality. Assigns status flag to each photon.
cf_flux_calibrate: Converts WEIGHT to ERG/CM2 for each photon.
cf_bad_pixels: Applies image-motion corrections to bad-pixel map.
cf_extract_spectra: Extracts LiF and SiC spectra only for target aperture. Wavelength array is user defined; default spacing is 0.013 Å.
Installing and Running the Pipeline
CalFUSE is available for machines running Solaris, Linux, and Mac OS X (10.2 and higher). We no longer have access to a Dec Alpha machine, so cannot support that platform.
Please see the document INSTALLING_CalFUSEv3.1.3 for installation instructions.
To run the code, the command is the same for both TTAG and HIST data:
prompt> calfuse P99901010011attagfraw.fit prompt> calfuse P99901010011ahistfraw.fit
The pipeline expects to find all data files (*raw.fit, *jitrf.fit, *hskpf.fit) in the current directory. It operates on one detector segment at a time.
CalFUSE v3.1 produces the following output files: an intermediate data file (IDF), discussed below; a bad-pixel map (BPM), with a format similar to the IDF; two extracted spectral files, one for each of the LiF and SiC channels; and (if IDL is installed on your machine) a pair of GIF or JPEG files, one an image of the detector and the other a count-rate plot.
The Intermediate Data File (IDF)
The photon-list files consist of three FITS binary tables. The first contains the photon events themselves. The pipeline does not discard any events, but uses a series of flags to indicate whether particular photons violate pulse-height limits, limb-angle constraints, etc. The IDL tool cf_edit (discussed below) allows users to modify these flags and combine IDF files from multiple exposures.
The second extension to the data files is a list of good-time intervals (GTI's). They are not used by the extraction routine, but may prove helpful to pipeline users.
The third extension is called a timeline table. For each second during the exposure, it lists a dozen parameters, including count rate, day-night status, and detector voltage. The pipeline uses this list to set the status flags for each photon.
Note: because of the file format employed by IDF files, extensions 1 and 3 must be read using the /fscale keyword.
idl> a=mrdfits('P99901010011attagfidf.fit',1,/fscale)
idl> help,a,/str
Elements of individual arrays must be addressed using the syntax
idl> print,a.time[3:30] -- not a[3:30].time
For details about the format and contents of the IDF, please see the document IDF_Format.txt.
CF_EDIT is an Interactive Data Language (IDL) visualization tool for the examination and modification of FUSE IDF files. Users without IDL licenses may use the IDL "Virtual Machine" version of the tool for free. Complete instructions for installing and using cf_edit are available from the FUSE IDL Tools web page. More information on the IDL Virtual Machine is available from RSI.
The Output Spectral File
The format of the output spectral files is somewhat changed from earlier versions of CalFUSE.
float WAVE Wavelength (Angstroms) float FLUX Flux (erg/cm2/s/A) float ERROR Gaussian errors (erg/cm2/s/A) int COUNTS Raw counts in extraction window float WEIGHTS Raw counts corrected for deadtime float BKGD Estimated background in extraction window short QUALITY Percentage of window used for extraction
Note that the QUALITY array no longer lists the number of Y pixels that contribute to each output X pixel. Instead, it is the fraction of the extraction window considered to contain valid data, expressed as a percentage (0 is all bad; 100 is all good).
Do-it-yourself-ers can use the following recipe to generate a flux-calibrated spectrum:
TARGET_COUNTS = WEIGHTS - BKGD TARGET_FLUX = TARGET_COUNTS * HC / LAMBDA / AEFF / EXPTIME /WPC
where AEFF = effective area in cm2, and WPC = width of output pixel in Ångstroms.
Additional Files Available from MAST
CalFUSE produces eight extracted spectral (*fcal.fit) files for each exposure. We combine them into a set of three observation-level files for submission to the MAST archive. Depending on the target and the questions that you are trying to answer, you may find that these files are of sufficient fidelity for scientific investigation. Here's a brief description of their contents:
ALL: For each channel (LiF 1A, SiC 1A, etc.), we combine data from all exposures in the observation into a single spectrum. If the individual spectra are bright enough, we cross correlate and shift before combining them. If the spectra are too faint, we combine the individual IDF files and extract a single spectrum to optimize the background model. The combined spectrum for each channel is stored in a separate extension.
Note: It turns out that the cataloging software used by MAST requires the presence of an ALL file for each exposure, not just for the entire observation. We now generate such a file, but it contains no data, only a FITS file header. The observation-level ALL files discussed above can be identified by the string "00000all" in their names.
ANO (all, night-only): Same format as the ALL files, but using only data obtained during the night-time portion of each exposure. These files are generated only for TTAG data, and only if EXPNIGHT > 0.
NVO (National Virtual Observatory): Contains a single spectrum spanning the entire FUSE wavelength range. The spectrum is assembled by cutting and pasting segments from the most sensitive channel at each wavelength. Segments are shifted to match LiF 1A between 1045 and 1070 Å. Columns are WAVE, FLUX, and ERROR and are stored in a single binary table extension.
Data-Analysis Tools
A variety of IDL routines to display and manipulate FUSE data are available from the FUSE IDL Tools Reference Page. Data-analysis tools distributed with the CalFUSE pipeline are reviewed in the document FUSE Tools in C. A subset of these programs, designed specifically for the manipulation of IDF files, is described in the IDF Cookbook, available in both PDF and HTML formats.
Bug Reports
Please report any anomalies to
fuse_support@pha.jhu.edu.
WebMasters: