JP-3235: Implement NSClean step for NIRSpec 1/f noise (spacetelescope…

…#8000)

CHANGES.rst

@@ -61,12 +61,26 @@ imprint
 - Updated the logging to report which imprint image is being subtracted from the
   science image. [#8041]
 
+nsclean
+-------
+
+- Implemented this new step, which is used to remove 1/f noise from NIRSpec
+  images. [#8000]
+
 photom
---------
+------
 
 - Added time-dependent correction for MIRI Imager data.
   [#8096, #8102, spacetelescope/stdatamodels#235]
 
+pipeline
+--------
+
+- Updated the ``calwebb_spec2`` pipeline to add in calling the ``nsclean`` step
+  for NIRSpec exposures. Also rearranged the order of the steps, so that
+  ``msa_flagging`` immediately follows ``assign_wcs``, so that both steps have
+  been applied before calling ``nsclean``. [#8000]
+
 pixel_replace
 -------------
 

CODEOWNERS.txt

@@ -24,6 +24,7 @@ jwst/ipc                 @philhodge         @stscirij
 jwst/jump                @dmggh             @hbushouse
 jwst/lastframe           @jemorrison        @dmggh
 jwst/linearity           @hbushouse         @stscirij
+jwst/nsclean             @hbushouse         @jemorrison
 jwst/outlier_detection   @stsci-hack        @jdavies-st
 jwst/persistence         @philhodge         @stscirij
 jwst/photom              @hbushouse         @dmggh

docs/jwst/nsclean/arguments.rst

@@ -0,0 +1,24 @@
+.. _nsclean_arguments:
+
+Step Arguments
+==============
+
+The ``nsclean`` step has the following optional arguments to control
+the behavior of the processing.
+
+``--mask_spectral_regions`` (boolean, default=True)
+  Mask regions in IFU and MOS images that are within the bounding boxes
+  for each slice or slitlet defined in the WCS object of the image.
+
+``--n_sigma`` (float, default=5.0)
+  The sigma-clipping threshold to use when searching for outliers
+  and illuminated pixels to be excluded from use in the fitting
+  process.
+
+``--save_mask`` (boolean, default=False)
+  A flag to indicate whether the mask constructed by the step
+  should be saved to a file.
+
+``--user_mask`` (string, default=None)
+  Path to a user-supplied mask file. If supplied, the mask is used
+  directly and the process of creating a mask in the step is skipped.

docs/jwst/nsclean/index.rst

@@ -0,0 +1,13 @@
+.. _nsclean_step:
+
+======================
+NSClean 1/f Correction
+======================
+.. toctree::
+   :maxdepth: 2
+
+   main.rst
+   reference_files.rst
+   arguments.rst
+
+.. automodapi:: jwst.nsclean

docs/jwst/nsclean/main.rst

@@ -0,0 +1,156 @@
+Description
+===========
+
+:Class: `jwst.nsclean.NSCleanStep`
+:Alias: nsclean
+
+Overview
+========
+The ``nsclean`` step applies an algorithm for removing correlated read
+noise from NIRSpec images. The noise often appears as faint vertical
+banding and so-called "picture frame noise." The algorithm uses dark
+(unilluminated) areas of an image to fit a background model in Fourier
+space. When the fit is subtracted, it removes nearly all correlated noise.
+Compared to simpler strategies, like subtracting a rolling median, this
+algorithm is more thorough and uniform. It is also computationally
+undemanding, typically requiring only a few seconds to clean a full-frame
+image.
+
+The correction can be applied to any type of NIRSpec exposure, including
+IFU, MOS, fixed slit, and Bright Object Time Series (BOTS), in both full-frame
+and subarray readouts. Time series (3D) data are corrected one integration
+at a time.
+
+.. note::
+
+   The step is currently not capable of processing images taken using the
+   "ALLSLITS" subarray. Other subarray types are allowed.
+
+Details on the source of the correlated noise and the algorithm used
+in the ``nsclean`` step to fit and remove it can be found in
+`Rauscher 2023 <https://ui.adsabs.harvard.edu/abs/2023arXiv230603250R/abstract>`_.
+
+Upon completion of the step, the step status keyword "S_NSCLEN" gets set
+to "COMPLETE" in the output science data.
+
+Assumptions
+===========
+As described below, the creation of a pixel mask depends on the presence
+of a World Coordinate System (WCS) object for the image, which is
+constructed by the :ref:`assign_wcs <assign_wcs_step>` step.
+In addition, creating a mask for IFU and MOS images depends on
+the presence of DQ flags assigned by the
+:ref:`msaflagopen <msaflagopen_step>` step.
+It is therefore required that those steps be run before attempting to
+apply ``nsclean``.
+
+Creation of an image mask
+=========================
+One of the key components of the correction is knowing which pixels are
+unilluminated and hence can be used in fitting the background noise.
+The step builds a mask on the fly for each image, which is used to mark
+useable and unuseable pixels. The mask is a 2D boolean array, having the same
+size as the image, with pixels set to True interpreted as being OK to use.
+The process of building the mask varies somewhat depending on the
+observing mode of the image being processed. Some features are common
+to all modes, while others are mode-specific. The following sections
+describe each type of masking that can be applied and at the end there
+is a summary of the types applied to each image mode.
+
+The user-settable step parameter `save_mask` can be used to save the
+mask to a file, if desired (see :ref:`nsclean step arguments <nsclean_arguments>`).
+
+Note that a user may supply their own mask image as input to the step,
+in which case the process of creating a mask is skipped. The step parameter
+`user_mask` is used to specify an input mask.
+
+IFU Slices
+----------
+For IFU images the majority of the mask is based on knowing which
+pixels are contained within the footprints of the IFU slices. To do
+this, the image's World Coordinate System (WCS) object is queried in
+order to determine which pixels are contained within each of the 30
+slices. Pixels within each slice are set to False (do not use) in the
+mask.
+
+MOS/FS Slits
+------------
+The footprints of each open MOS slitlet or fixed slit are flagged in
+a similar way as IFU slices. For MOS and FS images, the WCS object is
+queried to determine which pixels are contained within each open
+slit/slitlet and they are set to False in the mask.
+
+MSA Failed Open Shutters
+------------------------
+Pixels affected by stuck open MSA shutters are masked, because they
+may contain signal. This is accomplished by setting all pixels flagged by the
+:ref:`msaflagopen <msaflagopen_step>` step with DQ value "MSA_FAILED_OPEN"
+to False in the mask.
+
+NaN Pixels
+----------
+Any pixel in the input image that has a value of NaN is temporarily reset
+to zero for input to the fitting routine and flagged as False in the mask.
+Upon completion of the noise subtraction, this population of pixels is
+set back to NaN again in the output (corrected) image.
+
+Fixed-Slit Region Pixels
+------------------------
+Full-frame MOS and IFU images may contain signal from the always open
+fixed slits, which appear in fixed region in the middle of each image.
+The entire region containing the fixed slits is masked out when
+processing MOS and IFU images. The masked region is currently hardwired
+in the step to image indexes [1:2048, 923:1116], where the indexes are
+in x, y order and in 1-indexed values.
+
+Left/Right Reference Pixel Columns
+----------------------------------
+Full-frame images contain 4 columns of reference pixels on the left and
+right edges of the image. These are not to be used in the fitting
+algorithm and hence are set to False in the mask.
+
+Outliers
+--------
+Pixels in the unilluminated regions of the region can contain anomalous
+signal, due to uncaught Cosmic Rays, hot pixels, etc. A sigma-clipping
+routine is employed to find such outliers within the input image and set
+them to False in the mask. All pixels with values greater than
+:math:`median+n_sigma*sigma` are set to False in the mask.
+Here `median` and `sigma` are computed
+from the image using the astropy.stats `sigma_clipped_stats` routine,
+using the image mask to exclude pixels that have already been flagged
+and a clipping level of 5 sigma. `n_sigma` is a user-settable step
+parameter, with a default value of 5.0
+(see :ref:`nsclean step arguments <nsclean_arguments>`).
+
+Mode-Specific Masking Steps
+---------------------------
+The following table indicates which flavors of masking are applied to
+images from each type of observing mode.
+
+.. |c| unicode:: U+2713 .. checkmark
+
++--------------------------+-----+-----+-----+
+|                          |     | Mode|     |
++--------------------------+-----+-----+-----+
+| Masking                  | IFU | MOS |  FS |
++==========================+=====+=====+=====+
+| IFU Slices\ :sup:`1`     | |c| |     |     |
++--------------------------+-----+-----+-----+
+| Slits/Slitlets\ :sup:`1` |     | |c| | |c| |
++--------------------------+-----+-----+-----+
+| MSA_FAILED_OPEN          | |c| | |c| | |c| |
++--------------------------+-----+-----+-----+
+| NaN Pixels               | |c| | |c| | |c| |
++--------------------------+-----+-----+-----+
+| FS Region                | |c| | |c| |     |
++--------------------------+-----+-----+-----+
+| Reference Pix            | |c| | |c| | |c| |
++--------------------------+-----+-----+-----+
+| Outliers                 | |c| | |c| | |c| |
++--------------------------+-----+-----+-----+
+
+:sup:`1`\ The application of these steps can be turned on and off via
+the step parameter `mask_spectral_regions`. This parameter controls
+whether the "IFU Slices" and "Slits/Slitlets" portions of the masking
+are applied.

docs/jwst/nsclean/reference_files.rst

@@ -0,0 +1,3 @@
+Reference Files
+===============
+The ``nsclean`` step does not use any reference files.

docs/jwst/package_index.rst

@@ -41,6 +41,7 @@ Package Index
    model_blender/index.rst
    mrs_imatch/index.rst
    msaflagopen/index.rst
+   nsclean/index.rst
    outlier_detection/index.rst
    pathloss/index.rst
    persistence/index.rst

docs/jwst/pipeline/calwebb_spec2.rst

@@ -47,12 +47,14 @@ TSO exposures. The instrument mode abbreviations used in the table are as follow
 +==========================================================+=====+=====+=====+=====+=====+=====+=================+======+========+=====+
 | :ref:`assign_wcs <assign_wcs_step>`                      | |c| | |c| | |c| | |c| | |c| | |c| |       |c|       | |c|  |  |c|   | |c| |
 +----------------------------------------------------------+-----+-----+-----+-----+-----+-----+-----------------+------+--------+-----+
+| :ref:`msaflagopen <msaflagopen_step>`                    |     | |c| | |c| |     |     |     |                 |      |        |     |
++----------------------------------------------------------+-----+-----+-----+-----+-----+-----+-----------------+------+--------+-----+
+| :ref:`nsclean <nsclean_step>`                            | |c| | |c| | |c| |     |     |     |                 |      |        |     |
++----------------------------------------------------------+-----+-----+-----+-----+-----+-----+-----------------+------+--------+-----+
 | :ref:`imprint <imprint_step>`                            |     | |c| | |c| |     |     |     |                 |      |        |     |
 +----------------------------------------------------------+-----+-----+-----+-----+-----+-----+-----------------+------+--------+-----+
 | :ref:`background <background_step>`                      | |c| | |c| | |c| | |c| |     | |c| |       |c|       | |c|  |  |c|   |     |
 +----------------------------------------------------------+-----+-----+-----+-----+-----+-----+-----------------+------+--------+-----+
-| :ref:`msaflagopen <msaflagopen_step>`                    |     | |c| | |c| |     |     |     |                 |      |        |     |
-+----------------------------------------------------------+-----+-----+-----+-----+-----+-----+-----------------+------+--------+-----+
 | :ref:`extract_2d <extract_2d_step>`\ :sup:`1`            | |c| | |c| |     |     |     |     |                 | |c|  |  |c|   | |c| |
 +----------------------------------------------------------+-----+-----+-----+-----+-----+-----+-----------------+------+--------+-----+
 | :ref:`srctype <srctype_step>`\ :sup:`1`                  | |c| | |c| | |c| | |c| | |c| | |c| |       |c|       | |c|  |  |c|   | |c| |
@@ -181,12 +183,14 @@ abbreviations used in the table are as follows:
 +=======================================+============+==============+=================+==============+
 | :ref:`assign_wcs <assign_wcs_step>`   |   ALL      |     ALL      |       ALL       |       ALL    |
 +---------------------------------------+------------+--------------+-----------------+--------------+
+| :ref:`msaflagopen <msaflagopen_step>` |  MOS, IFU  |   MOS, IFU   |    MOS, IFU     |       MOS    |
++---------------------------------------+------------+--------------+-----------------+--------------+
+| :ref:`nsclean <nsclean_step>`         |  NONE      |    NONE      |      NONE       |      NONE    |
++---------------------------------------+------------+--------------+-----------------+--------------+
 | :ref:`imprint <imprint_step>`         |  NONE      |     IFU      |      NONE       |      NONE    |
 +---------------------------------------+------------+--------------+-----------------+--------------+
 | :ref:`background <background_step>`   |  NONE      |    NONE      |      NONE       |      NONE    |
 +---------------------------------------+------------+--------------+-----------------+--------------+
-| :ref:`msaflagopen <msaflagopen_step>` |  MOS, IFU  |   MOS, IFU   |    MOS, IFU     |       MOS    |
-+---------------------------------------+------------+--------------+-----------------+--------------+
 | :ref:`extract_2d <extract_2d_step>`   |  MOS, FS   |   MOS, FS    |    MOS, FS      |       MOS    |
 +---------------------------------------+------------+--------------+-----------------+--------------+
 | :ref:`srctype <srctype_step>`         |  NONE      |    NONE      |      NONE       |      NONE    |