CANICA data reduction with IRAF

Reduction of near infrared (NIR) data is laborious due to the amount of frames that are necessary to handle in order to produce an image of a single object. Typically around 10 frames would have to be handled in order to produce an image of net exposure time of 5 minutes in the K-band. The best way to handle these multiple exposure frames is through a pipeline. This manual describes such a pipeline, written for the reduction of data taken with CANICA, the CAnanea Near Infrared CAmera. The pipeline is a script based on various tasks under the IRAF environment.

What are the basic steps in NIR data Reduction?

Data Acquisition with CANICA and File name Nomenclature

CANICA acquires images under Linux environment. Images are acquired by sending an observing sequence containing RA-Dec coordinates and relative shifts through a Java script. Useful observational parameters, such as RA-Dec coordinates, filter, exposure time etc. of each image are written into image headers. The image names are formed from a root name by concatenating a three digit number, which is incremented sequentially until the sequence ends. Typical image names are of the type root_name001_I.fits, root_name002_I.fits, ...... root_name0nn_I.fits. Images taken using the Java sequence until now (June 2003) start with number 002 (001 is reserved for a "training" image). Bias subtraction is done during the acquisition. However for each image, a bias image is also stored. Bias images can be identified by their termination "_Z.fits".

Multiple exposure co-adds are not necessary with CANICA, given that the combination of parameters (image scale, Full well capacity and the sky background) allows single exposures of duration greater than 60 seconds in the K-band.

Basic function of the script

The most important step in the reduction of NIR images is the sky pattern subraction. sky in NIR images is a combination of the real sky background and a local background originated due to the emission of the warm optics surrounding the detector. The former is expected to be uniform, whereas the latter contribution can be non-uniform over the detector. Relative contribution of the two backgrounds depends on the wavelength. In addition, any vignetting gives a structure to the background. Hence it is necessary to subtract a sky frame from each object frame.

The pipeline starts by first identifying the sky frames in a given sequence. The RA-DEC coordinates are used to separate sky frames from object frames. Sky frames are then combined (imcombine combine=median) to prepare a star-free sky image.

The next step is to flat field the images using a previously prepared flat field frame. The thermal emission of the warm optics makes the task of getting a good flat field image somewhat difficult. Ideally flat field image should be prepared by subtracting a mean night sky image from a mean dawn/dusk image in the same filter, both exposed to same duration.

Centroid positions of a reference star are then obtained in each sky subtracted, flat-fielded image. These centroids are used to find shifts of each frame with respect to the reference frame. Images are aligned using the calculated shifts and then combined (imcombine combine=average, reject=minmax).

Photometry of the reference star (or any other object) is then carried out using the technique of aperture photometry. The combined image is then transformed into the normal astronomical coordinate convention (North to the top, east to the left).

CANICA script names and their usage

The script which does all the reduction is called canobj. For an error-free reduction canobj should be run twice. The first run is a trial run, basically to have a first-hand look at the images. In this run, an aproximate sky frame is prepared and subtracted from all the frames of the sequence. The resulting frames are displayed one by one, so as to enable a quick look at each frame. The display also helps to identify at least one star common to all the object frames. Before running canobj a second time, a script called canref is then run, which reads header parameters, and substitutes suitable values for the parameters of canobj. Now canobj can be re-run without the need to edit any of its parameters. Here is a summary of all the scripts:

Description of parameters of canobj

cl> epar canobj
                    Image Reduction and Analysis Facility
PACKAGE = user
   TASK = canobj
    
root    =                as23k  Root Image Name
imbeg   =                    1  First Image Number
imend   =                    6  Last Image Number
(sequenc=         as23k002.seq) Sequence file [file.seq/preview]
(skyfram=                stack) Sky frame [stack/name]
(flattyp=                 null) Flat-fielding strategy [skynew/skyold/flat/null]
(flatim =             ../flatk) Flat field frame Name
(refimag=          as23j002ref) Reference Image name
(iref   =                    2) ref star image number
(skyedit=                   no) Clean the sky using imedit?
(disp   =                  yes) Display before imcombine? 
(submed =                   no) Remove the horizontal band?
(immos  =                  yes) Mosaic the images?
(dophot =                  yes) Do photometry?
(xshift =                   2.) X-shift[=Xref-Xin] wrt refimag
(yshift =                 -15.) Y-shift[=Yref-Yin] wrt refimag
(skyout =                 600.) Minimum Distance of the off-field sky
(skyin  =                 150.) Minimum Distance of the on-field sky
(skysect=    [551:575,551:575]) Section for skyvalue estimation
(flatsec=    [551:575,551:575]) Section for flat normalization
(boxsize=                  15.) imalign boxsize
(bigbox =                  21.) imalign bigbox
(crval1 =            180.73875) RA of the header (in degrees)
(crval2 =      4.1288055555556) Dec of the header (in degrees)
(crpix1 =                512.5) X-Pixel corresponding to the RA-Dec
(crpix2 =                512.5) Y-Pixel corresponding to the RA-Dec
(cdelt1 =   8.8888888888889E-5) Pixel scale in RA (in degrees)
(cdelt2 =   8.8888888888889E-5) Pixel scale in Dec (in degrees)
(epoch  =                2000.) Epoch of RA-Dec
(dowcs  =                  yes) Transform the image to N_up E_left?
(xymag  =                   2.) XY reduction factor if dowcs=yes
(jhkzp  =    20.60 20.40 20.20) JHK Zeropoints
(mode   =                   ql)
Parameter Default/acceptable values Short description
root root file name (e.g. as23k for files of the type as23k002_I.fits, as23k003_I.fits, as23k004_I.fits etc)
imbeg the first number of the sequence (2 if the first image is as23k002_I.fits)
imend the last number of the sequence (7 if the last image is as23k007_I.fits)
sequenc preview/file.seq A filename (e.g. file.seq) containing the positions of a reference star or preview. The default option preview allows a trial run for the sequence. When sequenc=preview, the only parameters that are used are the first three parameters (root, imbeg and imend). So

cl> canobj sequenc=preview

can be run for any sequence without any prior knowledge of the kind of object it contains. The second run of canobj should have sequenc=file.seq, in which case many parameters of canobj should have been to set to proper values. The complementary script canref helps in setting these parameters. Read the help on canref. The file containing the positions of the reference star is automatically prepared when running canref. The file is a simple text file containing xy pixel coordinates of the chosen common star. In case, telescope pointing is wrong, it may be required to measure the position of the star in each frame, and then correct the entries in the sequence file. Any frame with coordinates outside [1,1024] pixel range implies that the frame is a sky frame. The file should contain exactly the same number of lines as the number of frames in the sequence.

skyfram stack/name Algorithm for determining the sky frames. With the option stack, sky frames are guessed based on the information in the sequence file sequenc, and then combined (imcombine combine=median zero=median) to prepare a star-free sky frame. Optionally the frame can be edited interactively using imedit (if skyedit=yes). all, sky frame is prepared using all the frames (suitable when the object is stellar, and observations are done with dithering technique).> The option name allows the user to give the name of an already prepared sky frame.
flattyp null / flat / skynew / skyold Flat-fielding strategies. With the option null flat fielding is not done. With flat, flat fielding is done using the flat image contained in the parameter flatim. With skynew, the normalized sky frame is used as a flat field image. In this case, sky value (not frame) is subtracted. With skyold, the sky image contained in the parameter flatim is used as a flat field image. Suitable when a good flat field is obtained in the current observing run using night-sky observations. When flattyp=skyold, sky frame formed using the images in the current sequence is used only to estimate the sky value, based on the parameter biassec. Effectively, this option reduces to the optical type of data reduction (i.e. flat fielding first, and then subtract an average sky value). It is not necessary to pre-normalize the flat fields, as the program normalizes the flat image using the average in the parameter flatsec.

Warning: flattyp=skynew and skyold are meant for test purposes. Hence values null or flat should be preferred for routine reductions.

flatim Flat field frame Name (when flattyp=flat) or the name of the previously prepared sky image (when flattyp=skyold) obtained in the current observing run using night-sky observations. In the former case, flat field image should have been prepared by subtracting a mean night sky image from a mean dawn/dusk image in the same filter, both exposed to same duration.
refimag as23j002ref Root name for files that contain the coordinate information of a common star in the reference frame. e.g. for refimag=as23j002ref, the working directory should contain files as23j002ref.coo (xy coordinates of the reference star), as23j002ref.fits (image containing the reference star at the reference position) and as23j002ref.xy (xy coordinates of stars for aperture photometry). These three files are automatically created during the run of canref. Otherwise should be created by the user. It is not necessary that the image refimag .fits be part of the current sequence. For example, it could be an image of the same field, but in another filter. See help on xshift,yshift.
iref 2 Reference star image number. To which image number, the reference position corresponds (e.g. 2 if the reference coordinates are measured in the image as23k002)? Its value is setup automatically during the run of canref. Otherwise should be entered by the user. It is an error if iref > imend or iref < imbeg.
skyedit yes/no Clean the sky using imedit? If skyedit=no, script uses the prepared sky (without displaying it) and continues with the rest of canobj. If yes, combined sky frame is displayed with the IRAF command imedit. Enter "q" to quit imedit and continue with canobj. Sky editing is required in only exceptional cases and hence normally enter "q" when the cursor is in the image window. Useful imedit keystroke commands are:
 a       Background replacement (rectangle)
 b       Background replacement (aperture)
 d       Constant value substitution (rectangle)
 e       Constant value substitution (aperture)
 u       Undo last change (see also 'i', 'j', and 'k')
 +       Increase radius by one
 -       Decrease radius by one
  Statistics
 q       Quit and save changes
 Q       Quit without saving changes
Read help on imedit for further details.
disp yes/no Display before imcombine? With disp=yes, bad images can be interactively rejected from the list of images to be aligned and averaged (combined). With disp=no, program runs in non-interactive mode.
submed no/yes Remove the horizontal band using the wrap technique? (default option "no" should be good for majority of the sequences, and need to be changed to "yes" in very exceptional cases).
immos yes/no Mosaic the images?
dophot yes/no Do photometry?
xshift,yshift 0,0 X-shift[=Xref-Xin] and Y-shift[=Yref-Yin] with respect to the reference image. Especially useful when all the images of a sequence need to be aligned to an image not included in the sequence (e.g. reference image is in the J-band, and we want all the K-images aligned to the J-band reference image). In such a case, mean shift in pixels between the two sets should be entered here. However, if coordinates of the reference star in each image are measured and entered into the "sequenc" file, then XY shifts should be left at their default values of zero.
skyout 600 Minimum shift of the off-field sky frame. If a frame in the sequence is shifted more than skyout pixels with respect to the reference frame, then it is assumed that the frame does not contain the object of interest, and that frame is used only as a sky frame.
skyin 150. Minimum shift of the on-field sky frame. While observing point sources, it is not necessary to go completely out of the object field to obtain good sky frames. It is a practice to center the point source in different quadrants for different exposures of a single sequence. The parameter skyin is the minimum shift of a frame with respect to the reference, in order to consider that frame suitable for sky extraction. So if the shift of a frame (wrt to the reference) is less than skyin, then it is not used for sky extraction, and if the shift is more than skyout, then it is not used for object extraction. All frames with shifts greater than skyin are used as sky frames. Frames in a sequence with shifts greater than skyin, but less than skyout, are used both for sky and object extraction.
Warning:: For off-field sky frames (shifts > skyout), make sure that each off-field sky in a sequence is distinct from the rest. Also when point sources are moved from one quandrant to the other, and shifts are intermediate between skyin and skyout, make sure that the position of the point source is not repeated in a single sequence. In both these cases, it is recommended to use cansky to get a good sky frame and then give it in the parameter skyfram.
skysect [551:575,551:575] Section for skyvalue estimation. This parameter is used only when flattyp=skynew. The section is used to estimate the average sky counts, which will be subtracted from all images.
flatsec [551:575,551:575] Section for flat normalization. Flat field frame is normalized to the average of this section.
boxsize 11 Imalign boxsize. The size in pixels of the box to use for the final centering, during which all the sources in refimag.coo are recentered in each image using the initial estimate of the relative shift for each image. Care should be taken to choose an appropriate value for this parameter, since it is highly data dependent. See imalign.
bigbox 15 Imalign bigbox. The size in pixels of the box to use for coarse centering. The coarse pass through the centering algorithm is made with the box centered at the nominal position of the first source in the coordinate list. Coarse centering is performed only if the shifts file is undefined. Care should be taken to choose an appropriate value for this parameter, since it is highly data dependent. Large values should be suspect until the final results are checked to see that the centering did not converge on the wrong coordinates, although the usual result for an inappropriate bigbox size is that the algorithm fails to converge and the task aborts. See imalign.

What to do if the task aborts?
The values of the parameters xshift,xshift,boxsize,bigbox are critical for a successful alignment of all the images in a sequence. If the telescope pointing and guiding is good, then imalign might be successful with the default values. However, if the centering algorithm fails, then task aborts, and canobj is terminated without completion. In such a case, you should rerun canobj after editing one or more of these four parameters. However it might take a lot of experience to tune the values of these parameters. Alternatively, sequenc=file.seq can be edited to put the exact centroid of the reference star in each image of a sequence. This can be achieved by displaying each sky subtracted image in the sequence and obtaining the centroid of reference star (with imexamin "a"). The latter option, though involves some interaction, is foolproof and is highly recommended whenever the task aborts.

crval1, crval2 0,0 Right Ascension and Declination (both in degrees) corresponding to the pixel (crpix1,crpix2). When canref is run, the header RA and Dec values of the reference frame are substituted here. The image header of the combined final image is updated with these keywords.
crpix1, crpix2 512.5,512.5 Pixel corresponding to the crval1, crval2 values. When canref is run, these parameters take the values for the center of the chip (canref.crpix=center) or the pixel value of the reference star (canref.crpix=refstar). The image header of the combined final image is updated with these keywords.
cdelt1, cdelt2 8.888889E-5, 8.888889E-5 Pixel scale in RA and Dec (in degrees/pixel). The default values correspond to 0.32 arcsec/pixel. It is assumed that the detector had been aligned before the observations, such that RA and Dec axes coincide with X-axis, and Y-axis respectively. Both the RA and Declination increase in the same direction as the pixels, i.e. North is up, and East is to the right.
dowcs yes Transform the image to N_up E_left? If yes, rotates the X-axis by 180 degrees, so that RA increases to the left (i.e. East is to the left).
xymag 2. XY reduction factor. If dowcs=yes, the image is compressed by the factor xymag, in addition to rotating the X-axis by 180 degrees.
jhkzp 20.60 20.40 20.20 JHK Zeropoints to be used when doing photometry of the objects in the file refimag.xy. The script automatically detects the filters and uses the corresponding Zeropoint. It is assumed that the the image header keyword FILTER2 = 1,2,3 for J,H and K bands respectively. For any other value of FILTER2, a zeropoint of 20 is used.

Description of parameters of canref

cl> epar canref
                    Image Reduction and Analysis Facility
PACKAGE = user
   TASK = canref

imref   =                    2  No of the ref image (2 for as40k002 etc.)
refcoo  =          as23j002ref  file (ref.coo) with ref star position or as40k00
(crpix  =               center) [center/refstar] XY pixels of RA-Dec
(mode   =                   ql)

Parameter Default/acceptable values Short description
imref 2 No of the ref image (2 for as40k002 etc.)
refcoo as23j002ref file (ref.coo) with ref star position or as40k002ref. File "ref.coo" should contain the position (pixels) of the reference star. Name of the file can be anything that ends with extension ".coo". If the file name does not end with ".coo", then it is assumed that you are giving the value suitable for canobj.refimag. Purpose of this option is to align images in a second filter (e.g. K) to previously analysed filter image (e.g. J). In this special case, files refimag.coo, refimag.xy and refimag.fits should have been created manually or through an earlier run of canref.
crpix center/refstar [center/refstar] XY pixels corresponding to the header RA-Dec. Default option corresponds to the center of the image [512.5,512.5]. With "refstar" header RA-Dec values are asigned to the reference star.

Guidelines and examples for successful running of canobj and canref

cl> ls as23*I.fits
as23h001_I.fits  as23h002_I.fits as23h003_I.fits 
as23h004_I.fits  as23h005_I.fits as23h006_I.fits  
as23j001_I.fits  as23j002_I.fits as23j003_I.fits
as23j004_I.fits  as23j005_I.fits as23j006_I.fits  
as23k001_I.fits  as23k002_I.fits as23k003_I.fits
as23k004_I.fits  as23k005_I.fits as23k006_I.fits
cl> unlearn canobj
cl> canobj as23j 1 6 sequenc=preview
... when images are displayed one by one, identify one star (preferably in image "as23j002_I.fits"), and find its centroid by pressing "a" when the cursor is the image window. Once the canobj trial run is over, edit (with any editor of your preference) a file called "ref.coo" and paste the xy pixel coordiantes of the identified star. You are now ready to run canref.
cl> canref 2 ref.coo 
...Hit "Return" for all the querries. You will note that the querried parameters are basically the first 3 parameters of canobj. When the task ends, the RA and Dec of each image in the sequence, as well as the shifts of images with respect to the reference image, appears on the screen. Canref passes all the parameters, that are necessary to align the images, to the parent task canobj. You are now ready to run canobj for the second time. If you have a good flat field, then change the canobj.flattyp=flat and canobj.flatim="flatname". Otherwise keep them at their default values.
cl> canobj 
...Hit "Return" for all the querries. Quit "imedit" and "imexamin" tasks that canobj invokes, with 'q", and if all the images look good, keep hitting "Return". A series of messages would appear on the text window. If the script ends successfully, the final rotated image (as23j_2a.fits) is displayed with the coordinate grid superimposed. On text screen, you will see magnitudes of the reference star in each image, followed by its magnitude in as23j_2.fits. Make sure that the coordinates and magnitudes of the star in different images of the sequence, do not differ by more than 2 pixels and 1 mag (approximately), respectively.

Input/Output filenames

Filename Examples Short description
root as23k An example for the root file name
iref 2 Reference image number. 2===> as23k002_I.fits is the reference image
rootiii_I.fits as23k002_I.fits, as23k003_I.fits, as23k004_I.fits etc Input file name with raw data (bias already subtracted during data acquisition)
rootiiisf.fits with iii = imbeg to imend as23k002sf.fits, as23k003sf.fits, as23k004sf.fits etc Sky subtracted, and flat-fielded Images. Saved only when the the script aborts due to error in centering stars.
rgrootiiisf.fits with iii = imbeg to imend rgas23k002sf.fits, rgas23k003sf.fits, rgas23k004sf.fits etc Sky subtracted, and flat-fielded and Aligned Images.
root_iref.fits as23k_2.fits The final image resulting from combining (imcombine combine=average reject=minmax) all rgrootiiisf.fits.
root_irefa.fits as23k_2a.fits X-axis of Image root_iref.fits rotated by 180 deg and compressed by xymag (generated only when dowcs=yes).
root_iref_sky.fits as23k_2_sky.fits Combined (and edited) sky frame (imcombine combine=median zero=median)
refimag.fits as23k002ref.fits Sky subtracted Image containing the reference star at the reference position.
root00iref.seq as23k002.seq Text file containing the xy pixels of the reference star in each of the sequence images.
refimag.coo as23k002ref.coo Text file containing the xy pixels of the reference star(s) in the reference image.
refimag.xy as23k002ref.xy Text file containing the xy pixels of all stars for which a quick photometry is to be carried out.
refimag.mag as23k002ref.mag Text file containing the photometric results of the first four stars in refimag.xy. Output of txdump.
rgrootiiisf.mag.1 with iii = imbeg to imend rgas23k002sf.mag.1, rgas23k003sf.mag.1, rgas23k004sf.mag.1 etc. Unedited output of photometry with phot.
root-canica.log as23k-canica.log Log file containing the details of reduction with canobj.
imexam.log Log file containing the results of imexamin task

Frequently Encountered Problems and Troubleshooting

Frequently asked questions (FAQs)


Written by Divakara Mayya (ydm@inaoep.mx). Last updated: 20 Jan 2004