Allow larger rtol on new test

[rosteen]

Realized this needed a slightly larger than default rtol when backporting, very small values ran into precision issues.

[pllim]

1E-5 still pretty small so probably ok but do we understand why the actual value deviates from the "truth" over time? Is someone recalibrating and re-uploading the data to Box?

[kecnry]

Do we know (or is the tolerance small enough that we don't care) what was different between between 4.0 and main that requires this?

[rosteen]

1E-5 still pretty small so probably ok but do we understand why the actual value deviates from the "truth" over time? Is someone recalibrating and re-uploading the data to Box?

This is a file I uploaded, I haven't changed it.

Do we know (or is the tolerance small enough that we don't care) what was different between between 4.0 and main that requires this?

Hm, it does actually seem to be some minor difference between 4.0 and main, not a precision/rounding thing, since the results are consistent across multiple runs within each branch. Is it worth investigating more deeply?

[pllim]

the results are consistent across multiple runs

But when did it start happening? Why do we only notice it now? 🧐

[rosteen]

the results are consistent across multiple runs

But when did it start happening? Why do we only notice it now? 🧐

I just added this test.

[pllim]

I don't understand... CI was passing an hour ago on main... Lemme rerun https://github.com/spacetelescope/jdaviz/actions/runs/12281137529

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 88.80%. Comparing base (cf9664f) to head (79c8442).
Report is 3 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3350   +/-   ##
=======================================
  Coverage   88.80%   88.80%           
=======================================
  Files         125      125           
  Lines       19155    19155           
=======================================
  Hits        17010    17010           
  Misses       2145     2145           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pllim]

Well, this is crazy, the rerun isn't failing on this test you are patching but some unrelated specviz2d URL open errors: https://github.com/spacetelescope/jdaviz/actions/runs/12281137529/job/34275135338

Are you sure this is deterministic?

[rosteen]

Well, this is crazy, the rerun isn't failing on this test you are patching but some unrelated specviz2d URL open errors: https://github.com/spacetelescope/jdaviz/actions/runs/12281137529/job/34275135338

Are you sure this is deterministic?

That's what I was saying to Kyle, the value this test produces on main is consistent (and passes the test, it's the value it's coded to check against). The value it consistently produces on 4.0.x is slightly different, just enough to fail the test. It's deterministic on both branches but slightly different.

[pllim]

Ahhh ok. Sorry, I misunderstood. So technically, rtol adjustment is not necessary for main then. In that case, you could adjust the expected value on main so it fails on main now but passes in backport branch with strict rtol, then from there you can bisect main. Hopefully that would smoke out the culprit but not always guaranteed.

[rosteen]

Closing this, I found an actual problem.