combining issues 726 and 736

[obi-wan76]

This PR has two components:

1. a small addition to match_data in order to set the correct pupil in certain situations. For example, WL images.
2. adding interpixel capacitance for both distortion extension in the simulated PSF, i.e., detector and oversample. For this, I moved the IPC effect from the gridded_library into webbpsf_core where we handle the IPC for the detector sampled PSF as it was originally.

Note that you can always check if IPC effects are included in your simulation by inspecting the header.

I tested this PR by using match_data for both the simple webbpsf simulation and a gridded simulation. Sample code:

file = 'jw04502071001_03104_00001_nrca3_cal.fits'
inst = webbpsf.setup_sim_to_match_file(file_input) 
sim_psf = inst.calc_psf( fov_arcsec = 3.153900235) # fov to match the size of the gridded PSF

c = webbpsf.gridded_library.CreatePSFLibrary(inst,inst.filter,  detectors = inst.detector, num_psfs = 1, psf_location = inst.detector_position )
grid = c.create_grid() # GriddedPSFModel object 

[mperrin]

@obi-wan76 did you see that the tests are failing on this PR? Specifically the function test_ipc_oversampling_equivalence. This was previously set up to test that you could either bin and then apply IPC, or apply IPC and then bin, and would get the same results in either order. Seems likely to me this is getting confused by the change to already have the IPC already added in the oversampled version, so it's getting applied twice in the test, or something like that.

[obi-wan76]

@mperrin thanks, I'll take a look at it when I come back next week.

[mperrin]

@obi-wan76 I'm looking at this PR but don't see any additions to match_data?

I just ran into a situation where I needed to improve that function to handle NIRCam bar coronagraph data keywords. I could add those lines of code to this PR, but I don't want to conflict with changes you may already have done?

[obi-wan76]

@mperrin I forgot to add the file to the push. It is showing now. It should handle coronagraph data keywords but please add any missing setup. Thanks!

[mperrin]

Thanks that change does what I needed :-)

(there may be an edge case still TBD for some of the dual channel coronagraph cases, using the SW channel with the LWB mask and so forth; but for right now those masks are treated interchangeably so this can be dealt with later)

[mperrin]

I added one additional enhancement to this, relevant for NIRCam bar mask sims:The match_data.setup_sim_to_match_file function will set the aperturename attribute. I added a few lines of code to use that aperture name value when looking up the offset along the bar occulter. This allows setup_sim_to_match_file to correctly setup a sim for cases in which the user used the Fiducial Point Override special requirement to override the default position on the coronagraph. (relevant for cycle 1 PIDs 1194 and 1412)

[mperrin]

To handle dual-channel NIRCam coronagraphy data is tricky, since the FITS header APERNAME keyword cannot be relied upon to correctly give the aperture name used. (Known issue, discovered during commissioning of the dual channel operations mode, not yet prioritized to be addressed in PPS and DMS). Thankfully @JarronL already wrote code to work around this in webbpsf_ext. With his permission I'm copying those files here into webbpsf so they can be used insetup_sim_to_match_file.

Example:

> nrc = webbpsf.setup_sim_to_match_file('jw01194002001_0310a_00001_nrca4_uncal.fits')
> nrc.aperturename
'NRCA4_400X256_MASKLWB_NARROW'

[obi-wan76]

I fixed a problem with the previous commit that the IPC effects were apply twice inside the gridded PSF path and also to apply IPC effects before the especial normalization inside gridded PSF. There is also a
new check in detectors.py so not to apply IPC effects twice in any of the calc_psf paths. I tested all the different combinations and they are all in agreement with previous results.
All distortion extension of calc_psf should have IPC by default.

[obi-wan76]

@Skyhawk172 can you take a look at this PR? I just fixed a problem with my previous commit. Thanks!

[Skyhawk172]

Ran a few cases along with the test mentioned above and everything runs successfully and as expected. Looks good!

[mperrin]

@obi-wan76 good morning. I've been reminded that this PR was approved but hasn't been merged yet, I guess perhaps because the tests were failing? Do you have a plan for looking into this, or would you like me to?

Do you know, does the fix you made to match_data in this address the case of NIRCam using the filters in the pupil wheel, for instance F323N as reported in issue #806 ?

[obi-wan76]

@mperrin the test that's failing is test_return_intermediates with KeyError: 'MASKSWB'. I get the following message:
ValueError: Filter MASKSWB does not have a defined nominal offset position along MASKSWB
I think the code should be picking the bar_offset from the filter but it is using MASKSWB and thus not finding the offset.
If I do the offset position by hand, e.g., nc.options['bar_offset'] = 0.5 then the test pass. However, this may be an indication of a larger configuration issue with MASKSWB.

[obi-wan76]

This PR also solves the problem from #806
For example:

file = './jw01059-o004_t029_nircam_f322w2-f323n_i2d.fits'
file_input = file
inst = webbpsf.setup_sim_to_match_file(file_input)

Setting up sim to match ./jw01059-o004_t029_nircam_f322w2-f323n_i2d.fits
iterating query, tdelta=3.0

MAST OPD query around UTC: 2022-03-14T18:59:16.344
                        MJD: 59652.79116138889

OPD immediately preceding the given datetime:
	URI:	 mast:JWST/product/R2022031405-NRCA3_FP1-1.fits
	Date (MJD):	 59652.6556
	Delta time:	 -0.1355 days

OPD immediately following the given datetime:
	URI:	 mast:JWST/product/R2022032002-NRCA3_FP1-0.fits
	Date (MJD):	 59655.2847
	Delta time:	 2.4936 days
User requested choosing OPD time closest in time to 2022-03-14T18:59:16.344, which is R2022031405-NRCA3_FP1-1.fits, delta time -0.136 days
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:JWST/product/R2022031405-NRCA3_FP1-1.fits to /Users/melendez/WFSC_Flight/webbpsf-data_1_2_1/webbpsf-data/MAST_JWST_WSS_OPDs/R2022031405-NRCA3_FP1-1.fits ... [Done]
Importing and format-converting OPD from /Users/melendez/WFSC_Flight/webbpsf-data_1_2_1/webbpsf-data/MAST_JWST_WSS_OPDs/R2022031405-NRCA3_FP1-1.fits
Backing out SI WFE and OTE field dependence at the WF sensing field point

Configured simulation instrument for:
    Instrument: NIRCam
    Filter: F323N
    Detector: NRCB5
    Apername: NRCB5_FULL
    Det. Pos.: (1024, 1024) 
    Image plane mask: None
    Pupil plane mask: None


psf = inst.calc_psf()

psf['DET_DIST'].header

PUPILINT= 'jwst_pupil_RevW_npix1024.fits.gz' / Pupil aperture intensity source  
PUPILOPD= 'OPD from supplied FITS HDUlist object' / Pupil OPD source            
OPD_FILE= 'R2022031405-NRCA3_FP1-1.fits' / Pupil OPD file name                  
OPDSLICE=                    0 / Pupil OPD slice number, if file is a datacube  
INSTRUME= 'NIRCam  '           / Instrument                                     
FILTER  = 'F323N   '           / Filter name                                    
EXTNAME = 'DET_DIST'           / This extension is oversampled.        

[pep8speaks]

Hello @obi-wan76, Thank you for updating !

• In the file webbpsf/__init__.py:

Line 95:1: E402 module level import not at top of file
Line 96:1: E402 module level import not at top of file
Line 97:1: E402 module level import not at top of file
Line 99:1: E402 module level import not at top of file
Line 114:1: E402 module level import not at top of file
Line 125:1: E402 module level import not at top of file
Line 127:1: E402 module level import not at top of file
Line 129:1: E402 module level import not at top of file
Line 131:1: E402 module level import not at top of file

• In the file webbpsf/detectors.py:

Line 37:29: E261 at least two spaces before inline comment
Line 37:30: E262 inline comment should start with '# '
Line 52:77: E261 at least two spaces before inline comment
Line 56:9: E266 too many leading '#' for block comment
Line 67:17: E225 missing whitespace around operator
Line 90:9: E265 block comment should start with '# '
Line 105:9: E722 do not use bare 'except'
Line 113:23: E127 continuation line over-indented for visual indent
Line 117:13: E265 block comment should start with '# '
Line 131:5: E303 too many blank lines (2)
Line 138:44: E251 unexpected spaces around keyword / parameter equals
Line 138:46: E251 unexpected spaces around keyword / parameter equals
Line 184:24: E225 missing whitespace around operator
Line 188:29: E225 missing whitespace around operator
Line 192:22: E221 multiple spaces before operator
Line 193:20: E221 multiple spaces before operator
Line 194:26: E225 missing whitespace around operator
Line 196:29: E225 missing whitespace around operator
Line 200:29: E225 missing whitespace around operator
Line 202:20: E221 multiple spaces before operator
Line 218:5: E303 too many blank lines (3)
Line 232:71: E225 missing whitespace around operator
Line 270:5: E303 too many blank lines (3)
Line 271:36: E226 missing whitespace around arithmetic operator
Line 271:39: E226 missing whitespace around arithmetic operator
Line 271:55: E226 missing whitespace around arithmetic operator
Line 271:58: E226 missing whitespace around arithmetic operator
Line 278:39: E226 missing whitespace around arithmetic operator
Line 278:47: E226 missing whitespace around arithmetic operator
Line 280:28: E222 multiple spaces after operator
Line 310:25: E226 missing whitespace around arithmetic operator
Line 310:28: E226 missing whitespace around arithmetic operator
Line 351:106: E251 unexpected spaces around keyword / parameter equals
Line 351:108: E251 unexpected spaces around keyword / parameter equals
Line 355:106: E251 unexpected spaces around keyword / parameter equals
Line 355:108: E251 unexpected spaces around keyword / parameter equals
Line 359:43: E226 missing whitespace around arithmetic operator
Line 421:32: E127 continuation line over-indented for visual indent

• In the file webbpsf/gridded_library.py:

Line 227:71: E114 indentation is not a multiple of four (comment)
Line 227:71: E116 unexpected indentation (comment)
Line 228:71: E114 indentation is not a multiple of four (comment)
Line 228:71: E116 unexpected indentation (comment)
Line 229:71: E114 indentation is not a multiple of four (comment)
Line 229:71: E116 unexpected indentation (comment)
Line 292:67: E226 missing whitespace around arithmetic operator
Line 304:69: E226 missing whitespace around arithmetic operator
Line 307:17: E303 too many blank lines (2)
Line 371:30: E225 missing whitespace around operator
Line 390:33: E231 missing whitespace after ','
Line 543:5: E731 do not assign a lambda expression, use a def
Line 566:23: E226 missing whitespace around arithmetic operator
Line 566:25: E226 missing whitespace around arithmetic operator
Line 567:28: E226 missing whitespace around arithmetic operator
Line 567:30: E226 missing whitespace around arithmetic operator
Line 568:23: E226 missing whitespace around arithmetic operator
Line 568:25: E226 missing whitespace around arithmetic operator
Line 569:23: E226 missing whitespace around arithmetic operator
Line 569:25: E226 missing whitespace around arithmetic operator
Line 570:23: E226 missing whitespace around arithmetic operator
Line 570:25: E226 missing whitespace around arithmetic operator
Line 572:27: E226 missing whitespace around arithmetic operator
Line 572:29: E226 missing whitespace around arithmetic operator
Line 572:32: E231 missing whitespace after ','
Line 573:27: E226 missing whitespace around arithmetic operator
Line 573:29: E226 missing whitespace around arithmetic operator
Line 573:32: E231 missing whitespace after ','
Line 575:48: E128 continuation line under-indented for visual indent
Line 576:48: E126 continuation line over-indented for hanging indent
Line 579:9: E303 too many blank lines (2)
Line 583:5: E303 too many blank lines (2)
Line 584:16: E226 missing whitespace around arithmetic operator
Line 589:31: E226 missing whitespace around arithmetic operator

• In the file webbpsf/match_data.py:

Line 1:1: E266 too many leading '#' for block comment
Line 26:38: E231 missing whitespace after ','
Line 37:17: E225 missing whitespace around operator
Line 37:46: E225 missing whitespace around operator
Line 40:20: E225 missing whitespace around operator
Line 46:20: E225 missing whitespace around operator
Line 49:51: E226 missing whitespace around arithmetic operator
Line 49:55: E226 missing whitespace around arithmetic operator
Line 53:5: E303 too many blank lines (2)
Line 61:76: E114 indentation is not a multiple of four (comment)
Line 61:76: E116 unexpected indentation (comment)
Line 74:37: E226 missing whitespace around arithmetic operator
Line 100:1: E303 too many blank lines (3)
Line 131:16: E225 missing whitespace around operator
Line 159:41: E226 missing whitespace around arithmetic operator
Line 161:22: E225 missing whitespace around operator
Line 165:58: E225 missing whitespace around operator
Line 210:52: E225 missing whitespace around operator
Line 210:75: E225 missing whitespace around operator

• In the file webbpsf/tests/test_detectors.py:

Line 151:1: E302 expected 2 blank lines, found 1
Line 151:47: E251 unexpected spaces around keyword / parameter equals
Line 151:49: E251 unexpected spaces around keyword / parameter equals
Line 169:81: E231 missing whitespace after ','
Line 188:24: E225 missing whitespace around operator
Line 194:24: E225 missing whitespace around operator

• In the file webbpsf/webbpsf_core.py:

Line 232:22: E701 multiple statements on one line (colon)
Line 311:26: E225 missing whitespace around operator
Line 350:36: E701 multiple statements on one line (colon)
Line 359:5: E303 too many blank lines (2)
Line 360:29: E127 continuation line over-indented for visual indent
Line 394:47: E114 indentation is not a multiple of four (comment)
Line 394:47: E116 unexpected indentation (comment)
Line 396:27: E701 multiple statements on one line (colon)
Line 397:39: E701 multiple statements on one line (colon)
Line 461:109: E202 whitespace before ')'
Line 476:33: E701 multiple statements on one line (colon)
Line 477:33: E701 multiple statements on one line (colon)
Line 479:72: W504 line break after binary operator
Line 480:37: W504 line break after binary operator
Line 489:43: E701 multiple statements on one line (colon)
Line 492:91: E701 multiple statements on one line (colon)
Line 493:92: E701 multiple statements on one line (colon)
Line 530:23: E126 continuation line over-indented for hanging indent
Line 620:48: E127 continuation line over-indented for visual indent
Line 711:29: E201 whitespace after '('
Line 726:1: E266 too many leading '#' for block comment
Line 800:63: E127 continuation line over-indented for visual indent
Line 801:63: E127 continuation line over-indented for visual indent
Line 802:63: E127 continuation line over-indented for visual indent
Line 803:63: E127 continuation line over-indented for visual indent
Line 861:9: E303 too many blank lines (3)
Line 916:5: E303 too many blank lines (2)
Line 945:18: E114 indentation is not a multiple of four (comment)
Line 945:18: E116 unexpected indentation (comment)
Line 976:5: E303 too many blank lines (2)
Line 984:54: E225 missing whitespace around operator
Line 1014:33: E261 at least two spaces before inline comment
Line 1151:87: E231 missing whitespace after ','
Line 1161:83: E231 missing whitespace after ','
Line 1166:53: E701 multiple statements on one line (colon)
Line 1169:9: E303 too many blank lines (2)
Line 1175:66: E251 unexpected spaces around keyword / parameter equals
Line 1175:68: E251 unexpected spaces around keyword / parameter equals
Line 1213:5: E303 too many blank lines (2)
Line 1236:31: E701 multiple statements on one line (colon)
Line 1237:31: E701 multiple statements on one line (colon)
Line 1241:76: E231 missing whitespace after ','
Line 1247:5: E303 too many blank lines (2)
Line 1247:28: E241 multiple spaces after ','
Line 1282:59: E222 multiple spaces after operator
Line 1283:39: E225 missing whitespace around operator
Line 1286:24: E221 multiple spaces before operator
Line 1299:32: E261 at least two spaces before inline comment
Line 1319:42: E222 multiple spaces after operator
Line 1340:31: E261 at least two spaces before inline comment
Line 1350:98: E225 missing whitespace around operator
Line 1391:27: E225 missing whitespace around operator
Line 1394:26: E225 missing whitespace around operator
Line 1397:33: E225 missing whitespace around operator
Line 1404:36: E261 at least two spaces before inline comment
Line 1409:43: E261 at least two spaces before inline comment
Line 1429:27: E226 missing whitespace around arithmetic operator
Line 1430:49: E225 missing whitespace around operator
Line 1437:5: E303 too many blank lines (2)
Line 1437:54: E226 missing whitespace around arithmetic operator
Line 1461:49: E251 unexpected spaces around keyword / parameter equals
Line 1461:51: E251 unexpected spaces around keyword / parameter equals
Line 1501:80: E251 unexpected spaces around keyword / parameter equals
Line 1501:82: E251 unexpected spaces around keyword / parameter equals
Line 1513:9: E265 block comment should start with '# '
Line 1520:34: E231 missing whitespace after ':'
Line 1520:46: E231 missing whitespace after ':'
Line 1520:57: E231 missing whitespace after ':'
Line 1520:88: E231 missing whitespace after ':'
Line 1521:19: E231 missing whitespace after ','
Line 1522:19: E231 missing whitespace after ','
Line 1523:19: E231 missing whitespace after ','
Line 1526:23: E701 multiple statements on one line (colon)
Line 1553:13: E303 too many blank lines (2)
Line 1586:53: E226 missing whitespace around arithmetic operator
Line 1586:74: E226 missing whitespace around arithmetic operator
Line 1589:23: E231 missing whitespace after ','
Line 1590:23: E231 missing whitespace after ','
Line 1591:23: E231 missing whitespace after ','
Line 1593:23: E231 missing whitespace after ','
Line 1593:86: E203 whitespace before ','
Line 1594:23: E231 missing whitespace after ','
Line 1595:23: E231 missing whitespace after ','
Line 1597:23: E231 missing whitespace after ','
Line 1598:23: E231 missing whitespace after ','
Line 1599:23: E231 missing whitespace after ','
Line 1601:23: E231 missing whitespace after ','
Line 1602:23: E231 missing whitespace after ','
Line 1603:23: E231 missing whitespace after ','
Line 1605:23: E231 missing whitespace after ','
Line 1606:23: E231 missing whitespace after ','
Line 1607:23: E231 missing whitespace after ','
Line 1661:5: E303 too many blank lines (2)
Line 1661:62: E251 unexpected spaces around keyword / parameter equals
Line 1661:64: E251 unexpected spaces around keyword / parameter equals
Line 1707:13: E731 do not assign a lambda expression, use a def
Line 1709:13: E731 do not assign a lambda expression, use a def
Line 1724:12: E225 missing whitespace around operator
Line 1731:9: E266 too many leading '#' for block comment
Line 1731:9: E303 too many blank lines (3)
Line 1740:35: E226 missing whitespace around arithmetic operator
Line 1740:76: E225 missing whitespace around operator
Line 1740:78: E226 missing whitespace around arithmetic operator
Line 1743:45: E251 unexpected spaces around keyword / parameter equals
Line 1743:47: E251 unexpected spaces around keyword / parameter equals
Line 1744:74: E226 missing whitespace around arithmetic operator
Line 1744:86: E226 missing whitespace around arithmetic operator
Line 1748:42: E251 unexpected spaces around keyword / parameter equals
Line 1748:44: E251 unexpected spaces around keyword / parameter equals
Line 1749:42: E251 unexpected spaces around keyword / parameter equals
Line 1749:44: E251 unexpected spaces around keyword / parameter equals
Line 1750:42: E251 unexpected spaces around keyword / parameter equals
Line 1750:44: E251 unexpected spaces around keyword / parameter equals
Line 1759:9: E266 too many leading '#' for block comment
Line 1760:9: E266 too many leading '#' for block comment
Line 1775:13: E303 too many blank lines (2)
Line 1793:13: E303 too many blank lines (2)
Line 1820:26: E222 multiple spaces after operator
Line 1822:33: E116 unexpected indentation (comment)
Line 1824:48: E261 at least two spaces before inline comment
Line 1946:45: E128 continuation line under-indented for visual indent
Line 2007:5: E303 too many blank lines (2)
Line 2035:5: E303 too many blank lines (2)
Line 2128:29: E701 multiple statements on one line (colon)
Line 2161:63: E225 missing whitespace around operator
Line 2169:67: E225 missing whitespace around operator
Line 2171:67: E225 missing whitespace around operator
Line 2174:67: E225 missing whitespace around operator
Line 2176:67: E225 missing whitespace around operator
Line 2205:21: E225 missing whitespace around operator
Line 2207:23: E225 missing whitespace around operator
Line 2209:23: E225 missing whitespace around operator
Line 2211:23: E225 missing whitespace around operator
Line 2237:22: E222 multiple spaces after operator
Line 2252:5: E303 too many blank lines (2)
Line 2270:34: E261 at least two spaces before inline comment
Line 2281:29: E231 missing whitespace after ','
Line 2287:19: E261 at least two spaces before inline comment
Line 2288:69: E226 missing whitespace around arithmetic operator
Line 2289:19: E225 missing whitespace around operator
Line 2296:21: E225 missing whitespace around operator
Line 2314:27: E225 missing whitespace around operator
Line 2332:22: E701 multiple statements on one line (colon)
Line 2430:80: W504 line break after binary operator
Line 2451:40: E225 missing whitespace around operator
Line 2480:57: E202 whitespace before ')'
Line 2494:25: E222 multiple spaces after operator
Line 2494:34: E261 at least two spaces before inline comment
Line 2510:67: E203 whitespace before ':'
Line 2513:65: E124 closing bracket does not match visual indentation
Line 2517:65: E124 closing bracket does not match visual indentation
Line 2521:65: E124 closing bracket does not match visual indentation
Line 2530:65: E128 continuation line under-indented for visual indent
Line 2541:62: E114 indentation is not a multiple of four (comment)
Line 2541:62: E116 unexpected indentation (comment)
Line 2546:9: E303 too many blank lines (2)
Line 2557:1: E302 expected 2 blank lines, found 1
Line 2579:33: E116 unexpected indentation (comment)
Line 2598:29: E701 multiple statements on one line (colon)
Line 2668:1: E303 too many blank lines (3)
Line 2820:62: W504 line break after binary operator
Line 2856:34: E261 at least two spaces before inline comment
Line 2860:26: E226 missing whitespace around arithmetic operator
Line 2927:61: E226 missing whitespace around arithmetic operator
Line 3032:37: E701 multiple statements on one line (colon)
Line 3043:32: E701 multiple statements on one line (colon)

Comment last updated at 2024-04-01 16:57:16 UTC

[mperrin]

@obi-wan76 I made a fix to the PR code so that the test in test_nircam passes, from my work on the part of this that was for issue 726.

But there's still several test failures from test_psfgrid, presumably from your work towards implementing 736. So those will still need to be addressed.

[codecov]

Codecov Report

Attention: Patch coverage is 28.08989% with 64 lines in your changes are missing coverage. Please review.

Project coverage is 60.08%. Comparing base (fcac7f1) to head (2c65265).

Files	Patch %	Lines
webbpsf/match_data.py	4.91%	58 Missing ⚠️
webbpsf/webbpsf_core.py	64.70%	6 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #768      +/-   ##
===========================================
- Coverage    60.55%   60.08%   -0.47%     
===========================================
  Files           16       16              
  Lines         6652     6732      +80     
===========================================
+ Hits          4028     4045      +17     
- Misses        2624     2687      +63     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mperrin]

@obi-wan76 I re-added the two commits that got lost in your rebase. And did a little bit of code cleanup (whitespace, etc) and one fairly minor revision to your distortion code: for the add_ipc_gridded variable that you added, since it was only needed internal to that one function it could just be a local variable. Does not need to be saved into the options dict for access anywhere else, as far as I could tell. Yes?

The tests all pass locally for me, but are failing on here still. This is pretty weird.

[mperrin]

Wait, no, I take that back: actually the test suite DOES fail for me locally, reproducing the error seen online. Hmm. Will investigate.