Flaky test failure: TestTx_multiple_simpleSend/adding_a_new_validator_catches_up: unexpected account balance

[mark-rushakoff]

The failure looks like:

        logger.go:257: time=2024-10-28T18:05:06.403Z level=INFO msg="Committed header" late_val_idx=2 root=gcosmos sys=engine e_sys=mirror m_sys=kernel height=1 hash=a4b9a14663564ab16687cb573b92e4b5eeec6489c2597f490c425581c6cf4f2f next_committing_height=2 next_committing_hash=d1e5bda310aae8ca065d2dbb5438dd6d58a7afc97ac9385560658dfd43fffb1d
        logger.go:257: time=2024-10-28T18:05:06.406Z level=INFO msg="Committed header" late_val_idx=2 root=gcosmos sys=engine e_sys=mirror m_sys=kernel height=2 hash=d1e5bda310aae8ca065d2dbb5438dd6d58a7afc97ac9385560658dfd43fffb1d next_committing_height=3 next_committing_hash=489371fcbcdcd1937a5c03607f04fa5799fd1438083a4019f06be22cafce5f22

        main_test.go:1155: response body for balance account 0: "{\"balance\":{\"denom\":\"stake\",\"amount\":\"10000\"}}"
        main_test.go:1158: 
            	Error Trace:	/home/runner/work/gcosmos/gcosmos/main_test.go:1158
            	Error:      	Not equal: 
            	            	expected: "9900"
            	            	actual  : "10000"
            	            	
            	            	Diff:
            	            	--- Expected
            	            	+++ Actual
            	            	@@ -1 +1 @@
            	            	-9900
            	            	+10000

Observed in https://github.com/gordian-engine/gcosmos/actions/runs/11559862928/job/32175473314, a build of main at commit e68d089.

A passing test looks like:

    logger.go:256: time=2024-10-28T14:16:24.310-04:00 level=INFO msg="Committed change to root store" late_val_idx=1 root=gcosmos serversys=driver height=2 apphash=450e950a7b9e54a0bb19a96d300f8cedbff34bd7efe059b2c480c3614f2f0112
    logger.go:256: time=2024-10-28T14:16:24.313-04:00 level=INFO msg="Committed change to root store" late_val_idx=1 root=gcosmos serversys=driver height=3 apphash=5bf6ad5f62e70b89cc8ef60723ea749f458901668b0105f4bad00df162d18b6e
    main_test.go:1155: response body for balance account 0: "{\"balance\":{\"denom\":\"stake\",\"amount\":\"9900\"}}"

Note that the passing test committed height 3 and the failing test only committed height 2. This sounds like an issue with the test, not a bug in production code.