DaaLoadTest_daa1 testIntrinsicPerformance addPackedDecimal: arrays first differed at element

[pshipton]

0.48 release stream

https://openj9-jenkins.osuosl.org/job/Test_openjdk11_j9_special.system_ppc64le_linux_Release_testList_2/71 - ubu22-ppc64le-1
DaaLoadTest_daa1_special_5m_8 -Xgcpolicy:gencon -Xshareclasses -Xjit -Xnocompressedrefs

22:48:54  DLT testFailure: testIntrinsicPerformance(net.openj9.test.arithmetics.TestPerformance): addPackedDecimal: arrays first differed at element [6]; expected:<-111> but was:<36>
22:48:54  DLT addPackedDecimal: arrays first differed at element [6]; expected:<-111> but was:<36>
22:48:54  DLT 	at org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:55)
22:48:54  DLT 	at org.junit.Assert.internalArrayEquals(Assert.java:532)
22:48:54  DLT 	at org.junit.Assert.assertArrayEquals(Assert.java:341)
22:48:54  DLT 	at net.openj9.test.arithmetics.TestPerformance.verify(TestPerformance.java:465)
22:48:54  DLT 	at net.openj9.test.arithmetics.TestPerformance.testAdd(TestPerformance.java:119)
22:48:54  DLT 	at net.openj9.test.arithmetics.TestPerformance.testIntrinsicPerformance(TestPerformance.java:76)

[github-actions]

Issue Number: 20258
Status: Open
Recommended Components: comp:gc, comp:test, comp:vm
Recommended Assignees: pshipton, jasonfengj9, keithc-ca

[pshipton]

12x x 5 grinder https://openj9-jenkins.osuosl.org/job/Grinder/3853/

1 repeat in https://openj9-jenkins.osuosl.org/job/Grinder_iteration_1/538 - cent9-ppc64le-1

10:34:13  DLT testFailure: testIntrinsicPerformance(net.openj9.test.arithmetics.TestPerformance): addPackedDecimal: arrays first differed at element [5]; expected:<0> but was:<1>
10:34:13  DLT addPackedDecimal: arrays first differed at element [5]; expected:<0> but was:<1>

Other failures in the grinder:
#20263
#20261 (comment)

[pshipton]

@hzongaro pls check if this needs to be resolved in 0.48 or moved out.

[hzongaro]

@dylanjtuttle, may I ask you to take a look at this? It might be good to determine when this problem first appeared, as well as figuring out what might be going wrong.

[hzongaro]

@dylanjtuttle - FYI that @IBMJimmyk is also looking at #20263 (comment). There are failures for this test with several distinct symptoms. It's not clear whether they represent one problem or many.

[dylanjtuttle]

So far I've only been able to reproduce the failure in #20263. I am running the test on ppc64le Ubuntu 20.04 on JDK 11. I will continue trying, and may try Ubuntu 22 if I continue to have problems.

[pshipton]

FYI the original failure was found on the 0.48 build.

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_special.system_ppc64le_linux_Personal_testList_4/93 - ubu22-ppc64le-2
DaaLoadTest_daa1_special_5m_9 -Xaggressive -Xgcpolicy:gencon -Xjit -Xnocompressedrefs

10:13:59  DLT testFailure: testIntrinsicPerformance(net.openj9.test.arithmetics.TestPerformance): Array access index out of bounds. convertIntegerToPackedDecimal is trying to access packedDecimal[0] to packedDecimal[5],  but valid indices are from 0 to -1.
10:13:59  DLT java.lang.ArrayIndexOutOfBoundsException: Array access index out of bounds. convertIntegerToPackedDecimal is trying to access packedDecimal[0] to packedDecimal[5],  but valid indices are from 0 to -1.
10:13:59  DLT 	at openj9.dataaccess/com.ibm.dataaccess.DecimalData.convertIntegerToPackedDecimal(DecimalData.java:277)
10:13:59  DLT 	at net.openj9.test.PDMoveShifts.TestShiftsAndConvert.testShiftRight(TestShiftsAndConvert.java:3307)
10:13:59  DLT 	at net.openj9.test.PDMoveShifts.TestShiftsAndConvert.testShiftRightLowerDstPrec(TestShiftsAndConvert.java:3360)

[dylanjtuttle]

I've run nearly 100 iterations of DaaLoadTest_daa1_special_5m_8 on Grinder with the 0.48 build on ppc64le, but I still can't seem to reproduce this failure. It's clearly possible, as shown by the links provided by Peter, so I need to figure out what I'm doing wrong. I want to try to and track down which commit is responsible for introducing the failure, but obviously I need to find a way to reproduce it with enough regularity that I would be able to tell at what commit it started appearing.

Given my lack of progress on this issue so far, it might be best to move this back to the 0.49 release.

[vij-singh]

Given the other DAA on P issues have been resolved (like #20263), does this one still occur or has it been resolved now?

[dylanjtuttle]

I've got a final Grinder run going atm with the new changes, I'll report the results back here when done.

[dylanjtuttle]

After some difficulties that required re-running the last grinder build, I can confirm that this symptom did not appear in 500 iterations. Oddly enough, I did get one instance of the segmentation error failure, which I thought wouldn't have been possible. The SDK used for the Grinder run was created with this job, which used the head of the 0.48 branches for OpenJ9 and OMR, so as long as the changes which fixed the other symptoms made it into the 0.48 branches by 10:24 AM EST on October 15th, they should have been included in this Grinder run.

[hzongaro]

Assuming we haven't seen any recent instances of this problem, I'm going to move this to the "Future" release. We can move to back to a planned release if the problem recurs.