From 956fd4e4eadf9def6064559c7a7e5ff6d90f490f Mon Sep 17 00:00:00 2001 From: Bryan Hilbert Date: Sun, 27 Jan 2019 10:39:42 -0500 Subject: [PATCH 1/5] Update index --- docs/index.rst | 61 ++++++++++++++++++++++++++++++++++++ docs/plug_in_user_module.rst | 2 ++ 2 files changed, 63 insertions(+) diff --git a/docs/index.rst b/docs/index.rst index 64495ee..cf69883 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -16,6 +16,67 @@ JWST_REFFILES: Reference File Generator Tool quickstart.ipynb api.rst +The **jwst_reffiles** package contains a framework to organize and automate the creation of calibration reference files for JWST instruments. These reference files are used by the `JWST calibration pipeline `_ for data calibration. + +There are three general stages within **jwst_reffiles** for reference file creation. + +1. :ref:`Identification and orgnization of input files ` +2. :ref:`Creation of an individual reference file ` +3. :ref:`Combining individual reference files into a final reference file ` + +In this documentation, we use the term *individual reference file* to refer to a reference file created from a single input file or group of files. For example, if a single dark current exposure can be used to create a readnoise image, then this readnoise image, onced saved in the appropriate format, is an *individual reference file*. Conversely, we refer to the mean readnoise image created from a stack of individual readnoise images as the *final reference file*. + + +**move the steps below to their own separate pages** + + +.. _organize_inputs: + +Identification and organization of input files +---------------------------------------------- + +The primary function performed by **jwst_reffiles** is to perform the organization and bookkeeping of input files to be used in the creation of reference files. The package is designed such that users can provide lists of input files (e.g. dark current files, flat field files), along with information on which types files should be used to create a given type of reference file, as well as some simple rules on which input files can be used in combination with others. **jwst_reffiles** then performs several tasks to prepare these input files for use in creating reference files. + +Group input files ++++++++++++++++++ + +Within a :ref:`configuration file `, the user can specify which combinations of inputs are needed to create a given type of reference file. For example, the user can specify that an individual gain reference file should be created using a pair of flat field files and a pair of dark current files. + +In addition, the user can define some simple rules for creating these groups of input files. Currently this includes a maximum time between files in the group. For example, to minimize any systematic variation in dark current, the user may specify that all dark current files in a single input group must have been observed within a 2 day time period. **Are there any other rules?** + +Using the grouping definitions and input file rules, **jwst_reffiles** will then create groups of input files, where each group can be used to create an :ref:`individual reference file `. + + +Determine which pipeline calibration steps must be run ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +Also within the main :ref:`configuration file `, the user can specify which calibration pipeline steps are necessary to be complete on the input files prior to creating individual reference files. **jwst_reffiles** will then examine the input files, determine which pipeline steps have already been completed, and which steps remain to be done. It then generates, for each file, the appropriate call to the JWST calibration pipeline such that the output will have all necessary steps completed. **jwst_reffiles** will run the pipeline commands, save the outputs, and use these in the input file groups. + + + +.. _individual_reffile: + +Create individal reference files +-------------------------------- + +Once the input files have been organized and calibrated to the requested level, **jwst_reffiles** will create an individual reference file from each group of input files. The software package used to create each type of reference file is specified by the user in the :ref:`main configuration file `. This package must be plugged in to **jwst_reffiles** in the manner described in the :ref:`plug-in ` page. + + + + + +.. _final_reffile: + +Create final reference files +---------------------------- + + + + +While **jwst_reffiles** provides the framework, user-generated packages can be plugged in + + + diff --git a/docs/plug_in_user_module.rst b/docs/plug_in_user_module.rst index 9668850..9f585ab 100644 --- a/docs/plug_in_user_module.rst +++ b/docs/plug_in_user_module.rst @@ -1,3 +1,5 @@ +.. _plugin: + How to Plug In A User-Generated Module ====================================== From 86d37b25f7a2aa286c21a42b602b56270a7382b0 Mon Sep 17 00:00:00 2001 From: Bryan Hilbert Date: Mon, 28 Jan 2019 15:18:22 -0500 Subject: [PATCH 2/5] Add basic documentation to help power users --- README.md | 7 + docs/configuration_files.rst | 337 +++++++++++++++++--- docs/final_reference_file_creation.rst | 10 + docs/index.rst | 60 +--- docs/individual_reference_file_creation.rst | 13 + docs/install.rst | 27 +- docs/organize_input_files.rst | 24 ++ 7 files changed, 384 insertions(+), 94 deletions(-) create mode 100644 docs/final_reference_file_creation.rst create mode 100644 docs/individual_reference_file_creation.rst create mode 100644 docs/organize_input_files.rst diff --git a/README.md b/README.md index 1109a5d..9c40b59 100755 --- a/README.md +++ b/README.md @@ -16,6 +16,13 @@ Then, install the package: cd jwst_reffiles pip install . ``` +or + +```python setup.py install``` + +## Documentation + +Documentation is still in the early stages of development, but is available on [ReadTheDocs](https://jwst-reffiles.readthedocs.io/en/latest/) ## Dependencies diff --git a/docs/configuration_files.rst b/docs/configuration_files.rst index 1aa6a76..d6d42ed 100644 --- a/docs/configuration_files.rst +++ b/docs/configuration_files.rst @@ -7,92 +7,92 @@ Jwst_reffiles makes use of multiple configuration files to control the behavior .. _main_cfg: -An example of the main configuraiton file: +An example of the main configuration file: -:: +.. parsed-literal:: - instrument: NIRCam - reflabels: [example_readnoise_module] - reftypes: [bpm,rdnoise,gain,linearity] + instrument_: NIRCam + reflabels_: [example_readnoise_module] + reftypes_: [bpm,rdnoise,gain,linearity] - default_reflabels: + default_reflabels_: bpm: bpm rdnoise: example_readnoise_module gain: gain_armin inputfiles: - dark_pattern: Bias|Dark|BIAS|DARK|ZERO - flat_pattern: GAIN|LIN + dark_pattern_: Bias|Dark|BIAS|DARK|ZERO + flat_pattern_: GAIN|LIN - basenamepattern: (.*)\_uncal\.fits$ + basenamepattern_: (.*)\_uncal\.fits$ # specificy the fits keywords to determine MJD time of # observation. They are automatically added to the list of required # fits keywords. if MJD-OBS does not exist, then DATE-OBS (and # TIME-OBS if specified) are used to determine the MJD. - dateobs_fitskey: DATE-END - timeobs_fitskey: TIME-END - mjdobs_fitskey: + dateobs_fitskey_: DATE-END + timeobs_fitskey_: TIME-END + mjdobs_fitskey_: - requiredfitskeys: [INSTRUME,DETECTOR,NGROUPS,NINTS,TGROUP,SUBARRAY] - optionalfitskeys: [FILTER,PUPIL] + requiredfitskeys_: [INSTRUME,DETECTOR,NGROUPS,NINTS,TGROUP,SUBARRAY] + optionalfitskeys_: [FILTER,PUPIL] - DD: - max_Delta_MJD: 2.0 + DD_: + max_Delta_MJD_: 2.0 - FF: - max_Delta_MJD: 2.0 - DDFF: - max_Delta_MJD: 10.0 + FF_: + max_Delta_MJD_: 2.0 + DDFF_: + max_Delta_MJD_: 10.0 - # reference file output basename: outrootdir[/outsubdir][/runID]/reflabel/reflabel[_outsubdir][_runID][.addsuffix].cmdID.reftype.fits + # reference file output basename_: outrootdir[/outsubdir][/runID]/reflabel/reflabel[_outsubdir][_runID][.addsuffix].cmdID.reftype.fits # if runID=AUTO, then the latest runID in the output dir outrootdir[/outsubdir] is taken output: - outrootdir: $JWST_MKREFS_OUTDIR - outsubdir: test1 - runID: AUTO + outrootdir_: $JWST_MKREFS_OUTDIR + outsubdir_: test1 + runID_: AUTO # how many leading zeros in runID - runIDNdigits: 3 - addsuffix: bla + runIDNdigits_: 3 + addsuffix_: bla # directory in which ssb products are saved. # if not specified, default is outrootdir[/outsubdir][/runID]/ssb # if skip_runID_ssbdir, then the runID is skipped - ssbdir: + ssbdir_: skip_runID_ssbdir: False # If ssblogFlag=True, the stdout from the strun command is saved in a log # file, same name as reduced filename, but with suffix log.txt - ssblogFlag: False + ssblogFlag_: False # If ssberrorlogFlag=True, the stderr from the strun command is # saved in a log file, same name as reduced filename, but with # suffix err.txt - ssberrorlogFlag: False + ssberrorlogFlag_: False # directories in which ssb products are looked for. comma-separated list! - pipeline_prod_search_dir: + pipeline_prod_search_dir_: batch: - batch_enabled: False - batchmode: Condor + batch_enabled_: False + batchmode_: Condor test1: 4 # values allowed: CRDS, SELF, filepattern - reffile4ssb: + reffile4ssb_: gain: CRDS bpm: CRDS rdnoise: CRDS # values allowed: CRDS, filename - validation: + validation_: gainreffile: CRDS bpmreffile: CRDS - example_readnoise_module: - reftype: rdnoise - imtypes: D - ssbsteps: dq_init + example_readnoise_module_: + reftype_: rdnoise + imtypes_: D + ssbsteps_: dq_init verbose: 3 boxsize: 128 sigma_threshold: 4 @@ -122,3 +122,264 @@ An example of the main configuraiton file: reffile4ssb: bpmreffile: /bla/bpm.fits + + + + +.. _instrument: + +Instrument +---------- + +The name of the instrument whose reference files are being created. Must be one of the JWST instruments. Capitalization is not important. + +.. _reflabels: + +Reference Labels +---------------- + +The reference labels are simply the names of the reference file creation packages to be run. + +.. _reftypes: + +Reference Types +--------------- + +List of the reference file types that may be created. At a minimum, this list must contain the reference file types corresponding to all of the entries in the reflabels_ list. + +.. _default_reflabels: + +Default Reference Labels +------------------------ + +In this section, the user can define which of the reference file creation packages (reflabels_) is the default for each reference type. + +.. _dark_pattern: + +Dark Pattern +------------ + +List of strings that will be used to identify dark current files within the list of input files. mkrefs will look for these strings in the filenames. + +.. _flat_pattern: + +Flat Pattern +------------ + +List of strings that will be used to identify flat field files within the list of input files. mkrefs will look for these strings in the filenames. + +.. _basenamepattern: + +Basename Pattern +---------------- + +String to use when looking for input files. + +.. _dateobs_fitskey: + +Date-Obs Fits Header Keyword +---------------------------- + +The header keyword in the input files that contains the date of the observation. Dates are needed to enforce rules when pairing darks and flats. For JWST, set this to **DATE-END**. + +.. _timeobs_fitskey: + +Time-Obs Fits Header Keyword +---------------------------- + +The header keyword in the input files that contains the time of the observation. Time are needed to enforce rules when pairing darks and flats. For JWST, set this to **TIME-END**. + +.. _mjdobs_fitskey: + +Obervation MJD Fits Header Keyword +---------------------------------- + +The header keyword in the input files that contains the time of the observation in MJD. + +.. _requiredfitskeys: + +Required Fits File Header Keywords +---------------------------------- + +List of header keywords which must be present in the input files. + +.. _optionalfitskeys: + +Optional FitsFile Header Keywords +--------------------------------- + +List of optional header keywords in the input files. Used for + +.. _DD: + +DD +--- + +Stands for dark dark, and indicates a situation where a pair of dark current ramps are needed to produce an individual reference file. + +.. _FF: + +FF +--- + +Stands for flat flat, and indicates a situation where a pair of flat field ramps are needed to produce an individual reference file. + +.. _DDFF: + +DDFF +---- + +Stands for dark dark flat flat, and indicates a situation where a pair of dark current ramps and a pair of flat field ramps are needed to produce an individual reference file. + +.. _max_Delta_MJD: + +Max_Delta_MJD +------------- +The maximum time allowed, in days, between input observations when creating pairs/groups of files. For example, to minimize the chances of dark current varying enough to impact reference file creation, you can set `max_Delta_MJD` to (e.g.) 2. In this case, when pairing dark current files (i.e. DD_) *jwst_reffiles* will not pair observations taken more than 2 days apart. + +.. _basename: + +Reference File Output Basename +------------------------------ + +Format of the output names for individual reference files. Output names will be automatically generated by *jwst_reffiles* to ensure accurate bookkeeping. The overall format of the reference file output names follows the convention: + +outrootdir_[/outsubdir_][/runID_]/reflabel/reflabel[_outsubdir_][_runID_][.addsuffix_].cmdID.reftype_.fits + +.. _outrootdir: + +Output Root Directory +--------------------- + +Path to the top level output directory for *jwst_reffiles*. Default is to define this within the **JWST_MKREFS_OUTDIR** environment variable, but any valid path is acceptable. + +.. _outsubdir: + +Output Subdirectory +------------------- + +Subdirectory name to add to the Output Root Directory when creating outputs. + +.. _runID: + +Run ID +------ + +An integer that will be used to create a unique subdirectory for *jwst_reffiles* outputs for a given "run" of the software. With this parameter, it is easy to organize the outputs and prevent conflicts for multiple runs using the same input files. Leaving this set to the default value of **AUTO** will cause *jwst_reffiles* to search for the most recent/highest existing run ID, and add 1 for the next run. + +.. _runIDNdigits: + +Number of Digits in the Run ID +------------------------------ + +The number of digits in the run ID. Leadind zeros are added as necessary. The default is 3. + +.. _addsuffix: + +Add Suffix +---------- + +Optional suffix that can be added to the output files from *jwst_reffiles*. + +.. _ssbdir: + +SSB Directory +------------- + +Directory where JWST calibration pipeline output files, which are often created in the process of running *jwst_reffiles*, will be saved. If this entry is left blank, the default value of outrootdir_[/outsubdir_][/runID_]/ssb/ will be used. + +.. _skip_runID_ssbdir: + +Skip RunID for SSB Directory +---------------------------- + +Boolean. If True, *jwst_reffiles* will not search for pipeline output files in the output directory for the current Run ID. + +.. _ssblogFlag: + +SSB Log Flag +------------ + +If ssblogFlag=True, the stdout from the JWST calibration pipeline call is saved in a log file, same name as pipeline output filename, but with suffix log.txt. + +.. _ssberrorlogFlag: + +SSB Error Log Flag +------------------ + +If ssberrorlogFlag=True, the stderr from the JWST calibration pipeline call is saved in a log file, same name as reduced filename, but with suffix err.txt. + +.. _pipeline_prod_search_dir: + +Pipeline Products Search Directories +------------------------------------ + +This is comma-separated list of directories. Prior to calling the calibration pipeline for a given input fits file, *jwst_reffiles* will search the directories listed here to see if the proper pipeline-processed file already exists. If blank, no searching will be done. + +.. _batch_enabled: + +Batch Enabled +------------- + +Boolean entry. If True, *jwst_reffiles* will be run in batch mode. (BATCH MODE NOT YET IMPLEMENTED) + +.. _batchmode: + +Batch Mode +---------- + +The batch system to use when running in batch mode. Default is Condor. + +.. _reffile4ssb: + +Reference Files for SSB +----------------------- + +In this section, list the types of reference files to use in the calls to the calibration pipeline. Options are **CRDS**, **SELF**, or a file pattern. If **CRDS** is used, then the appropriate reference files will be selected from the Calibration Reference Data System (`CRDS `_). This system contains only officially delivered reference files. + +If **SELF** is used, then calls to the calibration pipeline will use reference files generated from the current run of *jwst_reffiles*. Note that in this case, running *jwst_reffiles* becomes an iterative process. For example, run once to produce a superbias reference file. Then run again to use this superbias reference file to calibrate inputs when creating dark current reference files. + +Finally, you can enter a file pattern (e.g. /my/files/reffiles/gain/*gain.fits), in which case *jwst_reffiles* will use that file. + +.. _validation: + +Validation +---------- + +Can be either **CRDS** or a filename. + +.. _example_readnoise_module: + +Example Readnoise Module (example of a reference file creation module) +---------------------------------------------------------------------- + +This point in the config file is where you define the options that are specific to each reference file creation module. Any modules that are going to be run must be listed in the reflabels_ list. Any modules not in the reflabels_ list will be ignored. + +.. _reftype: + +Reference Type +-------------- + +Define the type of reference file created by this module. (e.g. gain, rdnoise) + +.. _imtypes: + +Image Types +----------- + +The type of inputs required by this package. (e.g. DD for a pair of dark ramps. FF for a pair of flat field ramps. DDFF for a pair of each.) + +.. _ssbsteps: + +SSB Steps +--------- + +Comma-separated list of calibration pipeline steps which must be complete on the input files prior to creating the reference file. Note that if the input files have been partially processed by the pipeline, the full list of completed steps must still be given here. For convenience, there is also a "-" shorthand that can be used. If the input files require all pipeline steps up to and including dark current subtraction, then you can enter "dark-". The pipeline steps currently recognized by *jwst_reffiles* includes all of those in the calwebb_detetor1 pipeline, and are called using the values from the list below. Other values will not be recognized. + +:: + group_scale, dq_init, saturation, ipc, firstframe, lastframe, superbias, refpix, linearity, persistence, rscd, dark_current, jump, rate. + +Example entries: + +dq_init, saturation, refpix +refpix- diff --git a/docs/final_reference_file_creation.rst b/docs/final_reference_file_creation.rst new file mode 100644 index 0000000..101f09a --- /dev/null +++ b/docs/final_reference_file_creation.rst @@ -0,0 +1,10 @@ +.. _final_reffile: + +Create final reference files +---------------------------- + +The creation of *final* reference files is similar to that of the *individual* reference files, where the user can plug in a module that takes as input a list of *individual* reference files and produces (and saves) a *final* reference file. + +Several basic and widely-uised combination methods (simple pixel-by-pixel mean, median, etc) will be included in the *jwst_reffiles* package for convenience. + + diff --git a/docs/index.rst b/docs/index.rst index cf69883..c2fc380 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -7,16 +7,19 @@ JWST_REFFILES: Reference File Generator Tool ============================================ .. toctree:: - :maxdepth: 2 + :maxdepth: 1 :caption: Contents: install.rst + organize_input_files.rst + individual_reference_file_creation.rst + final_reference_file_creation.rst plug_in_user_module.rst configuration_files.rst quickstart.ipynb api.rst -The **jwst_reffiles** package contains a framework to organize and automate the creation of calibration reference files for JWST instruments. These reference files are used by the `JWST calibration pipeline `_ for data calibration. +The **jwst_reffiles** package contains a framework to organize and automate the creation of calibration reference files for JWST instruments. These reference files are used by the `JWST calibration pipeline `_ for data calibration. There are three general stages within **jwst_reffiles** for reference file creation. @@ -27,59 +30,6 @@ There are three general stages within **jwst_reffiles** for reference file creat In this documentation, we use the term *individual reference file* to refer to a reference file created from a single input file or group of files. For example, if a single dark current exposure can be used to create a readnoise image, then this readnoise image, onced saved in the appropriate format, is an *individual reference file*. Conversely, we refer to the mean readnoise image created from a stack of individual readnoise images as the *final reference file*. -**move the steps below to their own separate pages** - - -.. _organize_inputs: - -Identification and organization of input files ----------------------------------------------- - -The primary function performed by **jwst_reffiles** is to perform the organization and bookkeeping of input files to be used in the creation of reference files. The package is designed such that users can provide lists of input files (e.g. dark current files, flat field files), along with information on which types files should be used to create a given type of reference file, as well as some simple rules on which input files can be used in combination with others. **jwst_reffiles** then performs several tasks to prepare these input files for use in creating reference files. - -Group input files -+++++++++++++++++ - -Within a :ref:`configuration file `, the user can specify which combinations of inputs are needed to create a given type of reference file. For example, the user can specify that an individual gain reference file should be created using a pair of flat field files and a pair of dark current files. - -In addition, the user can define some simple rules for creating these groups of input files. Currently this includes a maximum time between files in the group. For example, to minimize any systematic variation in dark current, the user may specify that all dark current files in a single input group must have been observed within a 2 day time period. **Are there any other rules?** - -Using the grouping definitions and input file rules, **jwst_reffiles** will then create groups of input files, where each group can be used to create an :ref:`individual reference file `. - - -Determine which pipeline calibration steps must be run -+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - -Also within the main :ref:`configuration file `, the user can specify which calibration pipeline steps are necessary to be complete on the input files prior to creating individual reference files. **jwst_reffiles** will then examine the input files, determine which pipeline steps have already been completed, and which steps remain to be done. It then generates, for each file, the appropriate call to the JWST calibration pipeline such that the output will have all necessary steps completed. **jwst_reffiles** will run the pipeline commands, save the outputs, and use these in the input file groups. - - - -.. _individual_reffile: - -Create individal reference files --------------------------------- - -Once the input files have been organized and calibrated to the requested level, **jwst_reffiles** will create an individual reference file from each group of input files. The software package used to create each type of reference file is specified by the user in the :ref:`main configuration file `. This package must be plugged in to **jwst_reffiles** in the manner described in the :ref:`plug-in ` page. - - - - - -.. _final_reffile: - -Create final reference files ----------------------------- - - - - -While **jwst_reffiles** provides the framework, user-generated packages can be plugged in - - - - - - Indices and tables ================== diff --git a/docs/individual_reference_file_creation.rst b/docs/individual_reference_file_creation.rst new file mode 100644 index 0000000..0468528 --- /dev/null +++ b/docs/individual_reference_file_creation.rst @@ -0,0 +1,13 @@ +.. _individual_reffile: + +Create individal reference files +-------------------------------- + +Once the input files have been :ref:`organized and calibrated ` to the requested level, **jwst_reffiles** will create an individual reference file from each group of input files. The software package used to create each type of reference file is specified by the user in the :ref:`main configuration file `, towards the bottom. An example, called :ref:`example_readnoise_module `, is shown in the linked configuration file. This package must be plugged in to **jwst_reffiles** in the manner described in the :ref:`plug-in ` page. + +Values for package-specific variables are provided by the user in this section of the configuration file as well. There are several inputs that are required for all reference file generation scripts. These are the :ref:`reftype `, :ref:`imtypes `, and :ref:`ssbsteps ` entries. In addition to these, any variables that are unique to the package can be listed here. In the :ref:`example_readnoise_module ` example, these include *verbose*, *boxsize*, and *sigma_threshold*. As described in the :ref:`Plug-in ` page, values entered here will be passed to the *example_readnoise_module* when it is called, unless they are overridden on the command line. + +For each reference file generation module specified in the configuration file's :ref:`reflabels ` section, the appropriate software will be called once for each of the :ref:`previously created ` group of pipeline-processed input files. Output files will be placed in the directory specified by the :ref:`reference file output directory ` in the configuration file. + +Note that each reference file creation module must take care of creating and saving the CRDS-`formatted `_ reference file itself. This means that the plugged-in module should use the appropriate `jwst` `data model `_, correctly populate all data and metadata, and save the file in the location provided by *jwst_reffiles*. + diff --git a/docs/install.rst b/docs/install.rst index 8935de7..fee42a7 100644 --- a/docs/install.rst +++ b/docs/install.rst @@ -1,5 +1,30 @@ Installing JWST_REFFILES ======================== +Note that *jwst_reffiles*' main dependency, the `JWST calibration pipeline `_ requires Python 3.5 or above. -Installation documentation goes here. +Environment Creation +-------------------- + +It may be easiest to create an environment to contain *jwst_reffiles* by first creating an environment around the JWST calibration pipeline. See the installation instructions in the `JWST repo `_ + +JWST_REFFILES installation +-------------------------- + +To download and install *jwst_reffiles*, clone from `GitHub `_. Make sure you are in your desired environment, and then install using setup.py. + +1. git clone https://github.com/spacetelescope/jwst_reffiles.git +2. cd jwst_reffiles +3. python setup.py install + + +Calling JWST_REFFILES +===================== + +Currently, *jwst_reffiles* is designed such that it can only be called from the command line. The top-level script to be called is `mkrefs.py`. Given a :ref:`configuration file `, as well as paths to collections of the necessary input files, `mkrefs.py` can be called in the following way: + +:: + + mkrefs.py -vvv --cfgfile main_config_file.cfg example_readnoise_module /path/to/dark/files/*DARK*_uncal.fits /path/to/flatfield/files/NRCN815A-LIN-53650[5678]*uncal.fits + +To call in this way, make sure that `mkrefs.py` is in your PYTHONPATH. Otherwise, prepend `python /path/to/mkrefs/` to the call above. The `-vvv` flag sets the verbosity. More v's leads to more details printed to the screen. `example_readnoise_module` is the reference file creation script that you wish to run. You can create more than one reference file type by providing a comma-separated list of reference file creation scripts. \ No newline at end of file diff --git a/docs/organize_input_files.rst b/docs/organize_input_files.rst new file mode 100644 index 0000000..9c55342 --- /dev/null +++ b/docs/organize_input_files.rst @@ -0,0 +1,24 @@ +.. _organize_inputs: + +Identification and Orgnization of Input Files +============================================= + +The primary function performed by **jwst_reffiles** is to perform the organization and bookkeeping of input files to be used in the creation of reference files. The package is designed such that users can provide lists of input files (e.g. dark current files, flat field files), along with information on which types files should be used to create a given type of reference file, as well as some simple rules on which input files can be used in combination with others. **jwst_reffiles** then performs several tasks to prepare these input files for use in creating reference files. + +.. _group_inputs: + +Group input files ++++++++++++++++++ + +Within a :ref:`configuration file `, the user can specify which combinations of inputs are needed to create a given type of reference file. For example, the user can specify that an individual gain reference file should be created using a pair of flat field files and a pair of dark current files. + +In addition, the user can define some simple rules for creating these groups of input files. Currently this includes a maximum time between files in the group. For example, to minimize any systematic variation in dark current, the user may specify that all dark current files in a single input group must have been observed within a 2 day time period. **Are there any other rules?** + +Using the grouping definitions and input file rules, **jwst_reffiles** will then create groups of input files, where each group can be used to create an :ref:`individual reference file `. + +.. _define_pipeline_steps: + +Determine which pipeline calibration steps must be run ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +Also within the main :ref:`configuration file `, the user can specify which calibration pipeline steps are necessary to be complete on the input files prior to creating individual reference files. **jwst_reffiles** will then examine the input files, determine which pipeline steps have already been completed, and which steps remain to be done. It then generates, for each file, the appropriate call to the JWST calibration pipeline such that the output will have all necessary steps completed. **jwst_reffiles** will run the pipeline commands, save the outputs, and use these in the input file groups. From 1b875441953ea8a3359dd8c314338adf61c1e89b Mon Sep 17 00:00:00 2001 From: Bryan Hilbert Date: Mon, 28 Jan 2019 15:49:58 -0500 Subject: [PATCH 3/5] Clarify some calling details --- docs/install.rst | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/install.rst b/docs/install.rst index fee42a7..021b86b 100644 --- a/docs/install.rst +++ b/docs/install.rst @@ -27,4 +27,10 @@ Currently, *jwst_reffiles* is designed such that it can only be called from the mkrefs.py -vvv --cfgfile main_config_file.cfg example_readnoise_module /path/to/dark/files/*DARK*_uncal.fits /path/to/flatfield/files/NRCN815A-LIN-53650[5678]*uncal.fits -To call in this way, make sure that `mkrefs.py` is in your PYTHONPATH. Otherwise, prepend `python /path/to/mkrefs/` to the call above. The `-vvv` flag sets the verbosity. More v's leads to more details printed to the screen. `example_readnoise_module` is the reference file creation script that you wish to run. You can create more than one reference file type by providing a comma-separated list of reference file creation scripts. \ No newline at end of file +To call in this way, make sure that `mkrefs.py` is in your PYTHONPATH. Otherwise, prepend `python /path/to/mkrefs/` to the call above. + +The `-vvv` flag sets the verbosity. More v's leads to more details printed to the screen. + +`example_readnoise_module` is the reference file creation script that you wish to run. You can create more than one reference file type by providing a comma-separated list of reference file creation scripts. + +You can provide as many input file lists as you need. *jwst_reffiles* combines the input file lists into a single list of files. It then examines each of the input files and determines what type of file it is (dark or flat. *jwst_reffiles* currently only supports these two file types.). \ No newline at end of file From 34230d580b4374d947734706a8b23ff852aa0a24 Mon Sep 17 00:00:00 2001 From: Bryan Hilbert Date: Mon, 28 Jan 2019 15:53:51 -0500 Subject: [PATCH 4/5] Add note that reflabels will soon disappear from config file --- docs/configuration_files.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/configuration_files.rst b/docs/configuration_files.rst index d6d42ed..3a7c440 100644 --- a/docs/configuration_files.rst +++ b/docs/configuration_files.rst @@ -138,7 +138,7 @@ The name of the instrument whose reference files are being created. Must be one Reference Labels ---------------- -The reference labels are simply the names of the reference file creation packages to be run. +The list of allowed reference labels. These are simply the names of the reference file creation packages that can be run. In the near future, this list will be generated automatically by searching for the wrapper scripts around plugged-in modules. At that time, reflabels will be removed from the config file. .. _reftypes: From 208fba5e5b51cd8b55de3c531423f39ba064adc4 Mon Sep 17 00:00:00 2001 From: Bryan Hilbert Date: Tue, 29 Jan 2019 16:54:42 -0500 Subject: [PATCH 5/5] Fix typos, respond to comments --- docs/configuration_files.rst | 29 +++++++++++++++-------------- docs/install.rst | 1 + docs/organize_input_files.rst | 2 +- 3 files changed, 17 insertions(+), 15 deletions(-) diff --git a/docs/configuration_files.rst b/docs/configuration_files.rst index 3a7c440..a2015bc 100644 --- a/docs/configuration_files.rst +++ b/docs/configuration_files.rst @@ -145,35 +145,35 @@ The list of allowed reference labels. These are simply the names of the referenc Reference Types --------------- -List of the reference file types that may be created. At a minimum, this list must contain the reference file types corresponding to all of the entries in the reflabels_ list. +List of the allowed reference file types that can be created. At a minimum, this list must contain the reference file types corresponding to all of the entries in the reflabels_ list. .. _default_reflabels: Default Reference Labels ------------------------ -In this section, the user can define which of the reference file creation packages (reflabels_) is the default for each reference type. +(NOT YET IMPLEMENTED) In this section, the user can define which of the reference file creation packages (reflabels_) is the default for each reference type. This is a convenience feature for the user. If you define `my_nircam_gain_script` as the default package to use for gain files, then you can call `mkrefs.py` from the command line and simply request `gain` rather than `my_nircam_gain_script`. .. _dark_pattern: Dark Pattern ------------ -List of strings that will be used to identify dark current files within the list of input files. mkrefs will look for these strings in the filenames. +List of strings that will be used to identify dark current files within the list of input files. mkrefs will look for these strings in the filenames. Note that the values shown in the example configuration file above are designed around ground testing files. JWST data obtained in flight will not have unique features in the filenames that identify them as darks, flats, etc. Future improvements to *jwst_reffiles* will address this. .. _flat_pattern: Flat Pattern ------------ -List of strings that will be used to identify flat field files within the list of input files. mkrefs will look for these strings in the filenames. +List of strings that will be used to identify flat field files within the list of input files. *jwst_reffiles* will look for these strings in the filenames. Note that the values shown in the example configuration file above are designed around ground testing files. JWST data obtained in flight will not have unique features in the filenames that identify them as darks, flats, etc. Future improvements to *jwst_reffiles* will address this. .. _basenamepattern: Basename Pattern ---------------- -String to use when looking for input files. +String to use when looking for input files. It may be that there are different versions (e.g. outputs from different points within the pipeline) of a file specified by the user in the input directory. If the basename pattern is set, *jwst_reffiles* will look only for the files matching that pattern. In the example above, we limit inputs to uncalibrated files. .. _dateobs_fitskey: @@ -187,28 +187,28 @@ The header keyword in the input files that contains the date of the observation. Time-Obs Fits Header Keyword ---------------------------- -The header keyword in the input files that contains the time of the observation. Time are needed to enforce rules when pairing darks and flats. For JWST, set this to **TIME-END**. +The header keyword in the input files that contains the time of the observation. Times are needed to enforce rules when pairing darks and flats. For JWST, set this to **TIME-END**. .. _mjdobs_fitskey: Obervation MJD Fits Header Keyword ---------------------------------- -The header keyword in the input files that contains the time of the observation in MJD. +The header keyword in the input files that contains the time of the observation in MJD. If specified, the value associated with this keyword is used to determinte the modified julian date (MJD) of the observation. .. _requiredfitskeys: Required Fits File Header Keywords ---------------------------------- -List of header keywords which must be present in the input files. +List of header keywords which must be present in the input files. Values from these keywords will be copied into the master table created by *jwst_reffiles*. If any input files do not contain all of these keywords, an error will be raised. .. _optionalfitskeys: Optional FitsFile Header Keywords --------------------------------- -List of optional header keywords in the input files. Used for +List of optional header keywords in the input files. Values from these keywords will be copied into the master table created by *jwst_reffiles*. If any of the keywords are missing in any of the input files, they are simply not copied to the master table, and *jwst_reffiles* proceeds without raising an error. .. _DD: @@ -242,10 +242,11 @@ The maximum time allowed, in days, between input observations when creating pair Reference File Output Basename ------------------------------ -Format of the output names for individual reference files. Output names will be automatically generated by *jwst_reffiles* to ensure accurate bookkeeping. The overall format of the reference file output names follows the convention: +Format of the output names for individual reference files. Output names will be automatically generated by *jwst_reffiles* to ensure accurate bookkeeping. The overall format of the reference file output names follows the convention shown below. **cmdID** is an ID assigned by *jwst_reffiles* for a particular run of the package. This helps to ensure unique file and directory names for the outputs. outrootdir_[/outsubdir_][/runID_]/reflabel/reflabel[_outsubdir_][_runID_][.addsuffix_].cmdID.reftype_.fits + .. _outrootdir: Output Root Directory @@ -272,7 +273,7 @@ An integer that will be used to create a unique subdirectory for *jwst_reffiles* Number of Digits in the Run ID ------------------------------ -The number of digits in the run ID. Leadind zeros are added as necessary. The default is 3. +The total number of digits in the run ID. Leading zeros are added as necessary. The default is 3. .. _addsuffix: @@ -293,7 +294,7 @@ Directory where JWST calibration pipeline output files, which are often created Skip RunID for SSB Directory ---------------------------- -Boolean. If True, *jwst_reffiles* will not search for pipeline output files in the output directory for the current Run ID. +Boolean. If True, the output directory (ssbdir_) for the pipeline output files will follow the convention outrootdir_[/outsubdir_]/ssb/. .. _ssblogFlag: @@ -335,7 +336,7 @@ The batch system to use when running in batch mode. Default is Condor. Reference Files for SSB ----------------------- -In this section, list the types of reference files to use in the calls to the calibration pipeline. Options are **CRDS**, **SELF**, or a file pattern. If **CRDS** is used, then the appropriate reference files will be selected from the Calibration Reference Data System (`CRDS `_). This system contains only officially delivered reference files. +NOT YET IMPLEMENTED. In this section, list the types of reference files to use in the calls to the calibration pipeline. Options are **CRDS**, **SELF**, or a file pattern. If **CRDS** is used, then the appropriate reference files will be selected from the Calibration Reference Data System (`CRDS `_). This system contains only officially delivered reference files. If **SELF** is used, then calls to the calibration pipeline will use reference files generated from the current run of *jwst_reffiles*. Note that in this case, running *jwst_reffiles* becomes an iterative process. For example, run once to produce a superbias reference file. Then run again to use this superbias reference file to calibrate inputs when creating dark current reference files. @@ -346,7 +347,7 @@ Finally, you can enter a file pattern (e.g. /my/files/reffiles/gain/*gain.fits), Validation ---------- -Can be either **CRDS** or a filename. +NOT YET IMPLEMENTED. Can be either **CRDS** or a filename. When comparing an output reference file to a previous version, this controls where the comparison file comes from. If set to **CRDS**, the most recent matching file in CRDS is used for comparison. If set to a filename, that file is used. .. _example_readnoise_module: diff --git a/docs/install.rst b/docs/install.rst index 021b86b..aeb654a 100644 --- a/docs/install.rst +++ b/docs/install.rst @@ -16,6 +16,7 @@ To download and install *jwst_reffiles*, clone from `GitHub