Review the value of tso3 spectral products - consider segmenting output x1dints files?

[stscijgbot-jp]

Issue JP-3841 was created on JIRA by Tyler Pauly:

JWST recently completed observations for two NIRSpec BOTS programs with total exposure times each greater than 50 hours (PIDs 5268 and 6491). These observations contain between 20,000 and 130,000 integrations, split across 18 - 24 segments.

Ops processing of these datasets through Level 2 pipelines has gone well, but saving the Level 3 outputs has caused issues. The initial troubleshooting found that saving a FITS file with 130,000 extensions causes file validation problems that may not hang, but also may take an unreasonable amount of time to complete. A simple solution is not readily apparent.

This ticket is to request a review of the products generated by the tso3 pipeline by the TSO WG, to understand if

• Users are utilizing these outputs
• If the existing algorithms are working as intended
• Possible effects of generating segmented level 3 products

The tso3 pipeline currently runs three steps for spectral data: outlier_detection, extract_1d, and white_light. The outlier_detection step runs on individual input segments and not on a collection of all inputs, meaning that outliers are detected on a per-segment basis. The extracted spectra from extract_1d will then differ from the spec2 products, even if we end up segmenting the tso3 outputs. The white_light output csv would not require segmenting, as its generation is not as taxing as the x1dints FITS file.