Failure image generated despite no change of code and their diff incorrectly rendered

[imaizume]

Hi, I'd like to report wired behavior.
I tried to take snapshot of root view for a simple ViewController.
After recorded its reference image successfully, turned recordMode into false and found the test failed.
Without any code nor visual changes to the ViewController, diff is rendered as below.

reference	failure	diff

• iOSSnapshotTestCase 6.1.0
• iPhone 8 Plus 12.4 Simulator
• Swift 4.2
• XCode Version 10.3 (10G8)

With iPhone Xs simulator, failure image slightly changes from its reference image (of course with no change in code nor storyboard) and diff is incorrectly rendered too.

reference	failure	diff

The ViewController

• loads its view from storyboard using https://github.com/tarunon/Instantiate (a wrapper for storyboard.instantiateInitialViewController())
• completes initialization for its view within viewDidLoad

I couldn't make sense of it and hope this to be resolved.

[mzhangFE]

I am struggling with the same issue here. After recorded the reference images successfully, turned recordMode into false. Without any code in the project, ran the tests and found the test failed. The images are shown as following:

The tests may all pass on some devices. On some other devices, the tests may fail or some of them fail while others succeed. For example, tests all pass on Simulator iPhone 8 with iOS 12.4 but all fail on Simulator iPhone 8 with iOS 11.4.

Any help would be much appreciated.

[reidmain]

There should not have been any changes that would cause a regression like this. Are you saying these snapshots worked correctly on a previous version of iOSSnapshotTestCase and 6.1.0?

If you are upgrading to a new iOS SDK version these sort of blips are typically expected because of the major changes that Apple typically makes to UIKit.

[zeksa90]

Same issue here. I am using :

• iOSSnapshotTestCase version 6.1.0 (did not upgrade from an earlier version)
• Xcode 11.1 (11A1027)
• macOS Catalina 10.15
• Simulator with iOS 13.1 (tried on all device types)
• fileNameOptions = [.screenSize]

As soon as I turn off recordMode, the test fails.

• Failure diff:
  

• Test fail message:
  failed - Snapshot comparison failed: Optional(Error Domain=FBSnapshotTestControllerErrorDomain Code=3 "Images different sizes" UserInfo={NSLocalizedFailureReason=referenceImage:{1242, 2688}, image:{414, 896}, FBDiffedImageKey=<UIImage:0x600002aa7f00 anonymous {1242, 2688}>, FBReferenceImageKey=<UIImage:0x600002aa74e0 anonymous {1242, 2688}>, FBCapturedImageKey=<UIImage:0x600002a83b10 anonymous {414, 896}>, NSLocalizedDescription=Images different sizes})

Through testing different cases I noticed that if fileNameOptions does not include .screenScale, the test will always fail.
For example: fileNameOptions = [.screenScale, .screenSize] will lead test to succeed.

[reidmain]

@zeksa90 that is a different bug that is potentially fixed by #100 and is in the v6.2.0 release.

You can see with your difference image that it recorded at 3x but then compared at 1x. It is a shortcoming of UIKit that if an image does not end with the appropriate @ suffix it assumes it to be 1x. The PR that fixed it makes iOSSnapshotTestCase use the actual pixel size rather than the logical device size.

[zeksa90]

Great @reidmain. Somehow I missed 6.2.0 release. I have tested v6.2.0 and bug from #100 is fixed.
Thanks

[reidmain]

@imaizume does v6.2.0 fix your issue?

[6ajlamyt2013]

Hi guys. I also caught this error, the error is most often seen on the screens with gradients , but visible differences I not see ...
environment
OSX 10.15.4
XCode 11.4
iOSSnapshotTestCase', '~> 6.2.0'
all simulators

Result after run test
reference _____________________ failure _________________________diff

Any help would be helpful , thanks

UPD
After the transition to the new version I not added the "perPixelTolerance" argument. After adding the tests have become more stable

FBSnapshotVerifyView(UIImageView(image: croppedImage),
                             perPixelTolerance: 0.05,
                             overallTolerance: 0.002)