OpenJ9 Perf Results with OJDK MHs enabled #12728

babsingh · 2021-05-19T19:21:16Z

This issue contains performance data for OpenJ9 with OJDK MHs enabled.

Target benchmarks

JRuby
Nashorn
scala-java-records-benchmark (https://github.com/harpocrates/scala-java-records-benchmark, java.lang.Record equality is much slower than HotSpot #12359)

The text was updated successfully, but these errors were encountered:

nbhuiyan · 2021-05-20T02:34:33Z

For reference, here are the recent results of jruby and nashorn benchmarks on JDK11 comparing the performance of OpenJ9 with the existing OpenJ9 MethodHandle implementation vs the reference OpenJDK implementation (RI). The diff (performance gap) is calculated using the following: abs(OpenJ9 - RI)/RI * 100.

Jruby

headius/bench2018

red-black (lower=better)

OpenJ9	RI	diff
1.9284	0.8336	56.77%

hash (higher = better)

	OpenJ9	RI	diff
bm_bighash	85.055	53.727	58.31%
bm_hash_long	54.22	55.655	2.58%
bm_hash_small2	188.857	184.896	2.14%
bm_hash_small4	146.211	137.465	6.36%
bm_hash_small8	93.796	76.529	22.56%

mandelbrot (lower=better)

OpenJ9	RI	diff
3.9179	1.2743	67.47%

kwargs (lower=better)

OpenJ9	RI	diff
2.7632	3.2769	18.59%

weird_sort (higher=better)

	OpenJ9	RI	diff
normal	15.363	90.886	83.1%
weird	10.117	14.225	28.88%

Nashorn (lower=better)

#5371

	OpenJ9	RI	diff
closureTest	144328.249	56421.089	155.81%
javaTest	697020.8	446143.001	56.23%
loadJS	696068092.611	539940873.905	28.92%
mathTest	191154280.676	43922475.21	335.21%
objectTest	329148.44	184116.502	78.77%
regexTest	34280233.376	30393064.373	12.79%

babsingh · 2021-05-20T12:15:35Z

These results are collected using scala-java-records-benchmark. They were collected on 19 May 21. Lower score refers to better performance. The unit of score is ns/op [time/operation]. OpenJ9 w/ OJDK MHs + JIT still underperforms. There are pending changes, which will eventually improve the performance.

Benchmark	OJDK JDK16	OpenJ9 JDK16	OpenJ9 w/ OJDK MHs	OJDK JDK16	OpenJ9 JDK16	OpenJ9 w/ OJDK MHs
	-Xint	-Xint	-Xint	With JIT	With JIT	With JIT
Bench.largeJavaEqualityTest	51392.904	10339.209	23231.945	15.687	206.891	20975.193
Bench.largeJavaInequalityTest	4770.602	1204.771	2356.366	5.297	47.955	2053.796
Bench.largeScalaEqualityTest	5049.162	1903.295	1910.793	50.379	49.278	42.252
Bench.largeScalaInequalityTest	1874.411	678.598	689.985	10.063	10.135	9.709
Bench.smallJavaEqualityTest	9199.715	2018.07	4463.956	5.676	47.39	3637.156
Bench.smallJavaInequalityTest	4750.179	1163.417	2363.743	5.167	40.946	2012.29
Bench.smallScalaEqualityTest	1029.631	403.254	409.576	10.491	16.898	16.152
Bench.smallScalaInequalityTest	771.266	311.534	317.501	4.673	3.789	3.747
------------------------------	----------	------------	------------------	----------	------------	------------------
Total time [s]	1016	1123	1149	943	1002	1011

babsingh · 2021-05-20T12:18:32Z

@DanHeidinga More data will be reported next week with changes that will lead to better performance.

0xdaryl · 2021-05-20T13:17:21Z

It is unlikely we will have results with JIT enabled until late next week at the earliest. Once @jdmpapin's inliner work is enabled we can re-bench, but more analysis will likely be needed to identify further opportuntities.

ThanHenderson · 2023-11-02T17:23:44Z

Nashorn JDK11

Setup

nashorn: downloaded from link at #5371
OpenJDK sha: d4a63e5e5c09a2d8fde2e2d8af2735e8cea8d27d
OpenJ9 sha: 4ba8f3d
OMR sha: 94e44382f4c519811ee1d45bda8601e1a8e79506
OpenJ9 builds: https://hyc-runtimes-jenkins.swg-devops.com/job/Pipeline-Build-Test-Personal/18880/
RI: https://jdk.java.net/java-se-ri/11-MR2

Raw values arens/op. diff% values are calculated by ((OpenJ9 with OJDK MHs) - Baseline) / Baseline * 100. Lower raw values are better; negative diff% values suggest better performance for OpenJ9 with OJDK MHs.

The sig diff? columns signify whether the 99.9% confidence intervals cross.

Results

AVG	OpenJ9	OpenJ9 w/ OJDK MHs	RI	OJDK,J9 diff%	OJDK,RI diff%
closeureTest	73840.444	86418.566	33505.513	17.03	157.92
javaTest	389427.281	394481.533	223702.7	1.3	76.34
loadJS	470714203.3	452817083.2	354069919.2	-3.8	27.89
mathTest	91896466.28	139162061.7	11701126.6	51.43	1089.3
objectTest	208376.025	197504.317	98528.471	-5.22	100.45
regexTest	21318446.5	19692941.02	19906138.23	-7.62	-1.07

ERROR				OJDK,J9 sig diff?	OJDK,RI sig diff?
closeureTest	421.577	1513.031	1177.102	TRUE	TRUE
javaTest	1773.48	15959.113	5953.699	FALSE	TRUE
loadJS	17884425.07	12030141.92	17208810.34	FALSE	TRUE
mathTest	400759.264	13710161.61	206223.082	TRUE	TRUE
objectTest	16014.034	6481.453	1282.917	FALSE	TRUE
regexTest	269241.567	515983.918	235774.228	TRUE	FALSE

babsingh · 2023-11-03T01:35:59Z

closeureTest and mathTest have huge regressions with OJDK MHs enabled in OpenJ9. Also, the errors are large for some benchmarks. We might have to collect perf profiles for OpenJ9 vs OpenJ9 w/ OJDK MHs enabled. Let's discuss next week once we have numbers from the other benchmarks.

ThanHenderson · 2023-11-04T14:44:25Z

JRuby JDK11

Setup

OpenJDK sha: d4a63e5e5c09a2d8fde2e2d8af2735e8cea8d27d
OpenJ9 sha: 4ba8f3d
OMR sha: 94e44382f4c519811ee1d45bda8601e1a8e79506
OpenJ9 builds: https://hyc-runtimes-jenkins.swg-devops.com/job/Pipeline-Build-Test-Personal/18880/
RI: https://jdk.java.net/java-se-ri/11-MR2
Benchmarks: https://github.com/headius/bench2018 with jruby-9.2.0.0 and the JDKs above, all other options set to empty.

diff% values are calculated by ((OpenJ9 with OJDK MHs) - Baseline) / Baseline * 100. For Higher is Better? == TRUE positive diff% values suggest better performance for OpenJ9 with OJDK MHs and the raw values are IPS (Iterations per second). For Higher is Better? == FALSE negative diff% values suggest better performance for OpenJ9 with OJDK MHs and the raw values are time in seconds.

run_all.rb was invoked 6 times. For tests reporting IPS, the harmonic mean was used to calculate the overall mean. For tests reporting time, the arithmetic mean was used to calculate the overall mean.

	Openj9	OpenJ9 w/ OJDK MHs	RI	OJDK,J9 diff%	OJDK,RI diff%	Higher is Better?
aref	23.52945362	24.50699197	36.62848373	4.15	-33.09	TRUE
kwargs	1.957237	1.848852225	1.665217558	-5.54	11.03	FALSE
mandelbrot	4.24541435	3.471999567	3.095899842	-18.22	12.15	FALSE
red_black	2.989383042	1.839528467	1.184561392	-7.53	55.29	FALSE

hash						TRUE
bm_bighash	159.1381292	130.014898	172.5808049	-18.3	-24.66
bm_long	75.61861284	72.55878889	83.83419221	-4.05	-13.45
bm_small2	196.5139465	196.8587689	307.9008781	0.18	-36.06
bm_small4	150.2621203	129.8387028	194.5893566	-13.59	-33.28
bm_small8	101.1572531	81.9990624	130.0253759	-18.94	-36.94

weird_sort						TRUE
normal	18.35367679	16.1438805	33.99895687	-12.04	-52.52
weird	8.499955682	9.731477391	11.68828367	14.49	-16.74

ThanHenderson · 2023-12-11T17:56:28Z

Evaluation after OJDK MH backport to JDK 11 and 8

Setup

JDK8

OpenJDK sha: 9a702f6d35a13affd72c1b996ca991ae40dac100
OpenJ9 sha: 4ba8f3d
OMR sha: 69bd2849661cd4225367de749505c4bc62ac4244
OpenJ9 builds: https://hyc-runtimes-jenkins.swg-devops.com/job/Build_JDK8_x86-64_linux_Personal/2945/
RI: https://jdk.java.net/java-se-ri/8-MR5

JDK11

OpenJDK sha: f229543b686dd32abe2b3f5565a46df0d70414b9
OpenJ9 sha: 58dc933
OMR sha: 69bd2849661cd4225367de749505c4bc62ac4244
OpenJ9 builds: https://hyc-runtimes-jenkins.swg-devops.com/job/Build_JDK11_x86-64_linux_Personal/5956/
RI: https://jdk.java.net/java-se-ri/11-MR2

JDK17

#OpenJDK sha: 549540814d003296474c332e25c5f109ca218cd2
OpenJ9 sha: 21f742f
OMR sha:137778af49aa93a5780b8eebb9d688f4be8e47ac
OpenJ9 builds: https://hyc-runtimes-jenkins.swg-devops.com/job/Build_JDK17_x86-64_linux_Personal/1773/
RI: https://jdk.java.net/java-se-ri/17# JDK17

JRuby

Benchmarks: https://github.com/headius/bench2018 with jruby-9.2.0.0 and the JDKs above, all other options set to empty.

diff% values are calculated by ((OpenJ9 with OJDK MHs) - Baseline) / Baseline * 100. For Higher is Better? == TRUE positive diff% values suggest better performance for OpenJ9 with OJDK MHs and the raw values are IPS (Iterations per second). For Higher is Better? == FALSE negative diff% values suggest better performance for OpenJ9 with OJDK MHs and the raw values are time in seconds.

run_all.rb was invoked 3 times. For tests reporting IPS, the harmonic mean was used to calculate the overall mean. For tests reporting time, the arithmetic mean was used to calculate the overall mean.

JRuby: JDK8

	OpenJ9	OpenJ9 (OJDK MHs)	RI	OJDK MHs,J9 diff%	OJDK MHs,RI diff%	Higher is Better?
aref	25.449	22.904	36.198	-10	-36.72	TRUE
kwargs	1.8678	1.9312	1.6136	3.39	19.69	FALSE
mandelbrot	4.2291	3.3686	2.7213	-20.35	23.79	FALSE
red_black	2.0926	1.8215	1.1107	-12.95	64	FALSE
hash						TRUE
bm_bighash	152.31	159.80	224.42	4.92	-28.79
bm_long	72.250	75.542	89.592	4.56	-15.68
bm_small2	186.38	235.23	384.75	26.21	-38.86
bm_small4	143.63	177.04	283.62	23.26	-37.58
bm_small8	89.667	98.510	165.02	9.86	-40.31
weird_sort						TRUE
normal	19.865	16.857	32.185	-15.14	-47.63
weird	7.9614	9.9400	11.120	24.85	-10.62

JRuby: JDK11

	OpenJ9	OpenJ9 (OJDK MHs)	RI	OJDK MHs,J9 diff%	OJDK MHs,RI diff%	Higher is Better?
aref	23.948	24.851	32.318	3.77	-23.1	TRUE
kwargs	1.8752	1.8618	1.6800	-0.71	10.82	FALSE
mandelbrot	4.1803	3.3257	3.1520	-20.44	5.51	FALSE
red_black	1.8992	1.8616	1.1793	-1.98	57.85	FALSE
hash						TRUE
bm_bighash	160.82	137.41	177.96	-14.55	-22.78
bm_long	74.798	74.146	84.290	-0.87	-12.03
bm_small2	190.76	217.21	312.24	13.86	-30.43
bm_small4	140.41	146.92	192.44	4.64	-23.65
bm_small8	90.429	88.623	130.46	-2	-32.07
weird_sort						TRUE
normal	15.832	17.277	32.451	9.13	-46.76
weird	8.2806	9.4855	11.605	14.55	-18.26

JRuby: JDK17

	OpenJ9 (OJDK MHs)	RI	diff%	Higher is Better?
aref	23.937	39.001	-38.62	TRUE
kwargs	1.9658	1.5349	28.08	FALSE
mandelbrot	3.9489	2.7269	44.81	FALSE
red_black	2.0294	1.1698	73.47	FALSE
hash				TRUE
bm_bighash	156.52	197.97	-20.94
bm_long	75.412	87.625	-13.94
bm_small2	224.50	298.25	-24.73
bm_small4	192.11	210.06	-8.55
bm_small8	120.63	131.68	-8.39
weird_sort				TRUE
normal	18.674	36.577	-48.95
weird	8.8396	13.259	-33.33

Nashorn

nashorn: downloaded from link at #5371

Raw values arens/op. diff% values are calculated by ((OpenJ9 with OJDK MHs) - Baseline) / Baseline * 100. Lower raw values are better; negative diff% values suggest better performance for OpenJ9 with OJDK MHs.

The sig diff? columns signify whether the 99.9% confidence intervals cross.

Nashorn: JDK8

AVG	OpenJ9	OpenJ9 (OJDK MHs)	RI	OJDK MHs,J9 diff%	OJDK MHs,RI diff%
closeureTest	91505.784	93466.824	54333.724	2.14	72.02
javaTest	560854.387	563618.939	351041.973	0.49	60.56
loadJS	316862499.5	336529749.7	313854511.7	6.21	7.22
mathTest	73244139.44	76200520.01	22154277.01	4.04	243.95
objectTest	161927.127	149042.848	64201.087	-7.96	132.15
regexTest	20030463.34	17915025.36	23130880.73	-10.56	-22.55

ERROR				OJDK MHs/J9 sig diff?	OJDK MHs/RI sig diff?
closeureTest	776.574	1307.805	1020.998	FALSE	TRUE
javaTest	7143.667	7581.127	7808.42	FALSE	TRUE
loadJS	7894732.838	11297730.9	18182676.45	TRUE	FALSE
mathTest	1480766.703	1069644.212	660128.421	TRUE	TRUE
objectTest	2467.68	2115.606	1193.525	TRUE	TRUE
regexTest	172951.024	252756.481	251662.393	TRUE	TRUE

Nashorn: JDK11

AVG	OpenJ9	OpenJ9 (OJDK MHs)	RI	OJDK MHs,J9 diff%	OJDK MHs,RI diff%
closeureTest	76699.613	79532.282	38031.226	3.69	109.12
javaTest	394880.844	388692.038	231436.955	-1.57	67.95
loadJS	307194678.5	299091809.6	243569415.8	-2.64	22.8
mathTest	74596025.51	75703800.27	17827959.42	1.49	324.64
objectTest	204520.483	175946.441	100407.056	-13.97	75.23
regexTest	20824044.65	19494917.04	19827037.09	-6.38	-1.68

ERROR				OJDK MHs/J9 sig diff?	OJDK MHs/RI sig diff?
closeureTest	1173.467	1261.971	2431.991	TRUE	TRUE
javaTest	16818.449	8550.381	2424.928	FALSE	TRUE
loadJS	9896791.459	9993521.638	12978266.56	FALSE	TRUE
mathTest	1445430.885	3010597.129	278380.344	FALSE	TRUE
objectTest	8965.226	2438.185	995.478	TRUE	TRUE
regexTest	385095.336	587363.531	206990.58	TRUE	FALSE

Nashorn: JDK17

AVG	OpenJ9 (OJDK MHs)	RI	J9,RI diff%
closeureTest	77230.321	31929.544	141.88
javaTest	406561.527	210242.633	93.38
loadJS	317260120.4	190994449.7	66.11
mathTest	82003116.31	16345537	401.69
objectTest	209020.914	101615.085	105.7
regexTest	20157479.69	17433295.22	15.63

ERROR			J9/RI sig diff?
closeureTest	991.429	1548.524	TRUE
javaTest	9544.566	3124.021	TRUE
loadJS	8859952.78	11497125.36	TRUE
mathTest	3268697.686	480921.858	TRUE
objectTest	11150.499	1281.714	TRUE
regexTest	189672.389	328671.301	TRUE

ThanHenderson · 2023-12-12T16:39:01Z

DayTrader7

Setup

JDK8

OpenJDK sha: 9a702f6d35a13affd72c1b996ca991ae40dac100
OpenJ9 sha: 4ba8f3d
OMR sha: 69bd2849661cd4225367de749505c4bc62ac4244
OpenJ9 builds: https://hyc-runtimes-jenkins.swg-devops.com/job/Build_JDK8_x86-64_linux_Personal/2945/

JDK11

OpenJDK sha: f229543b686dd32abe2b3f5565a46df0d70414b9
OpenJ9 sha: 58dc933
OMR sha: 69bd2849661cd4225367de749505c4bc62ac4244
OpenJ9 builds: https://hyc-runtimes-jenkins.swg-devops.com/job/Build_JDK11_x86-64_linux_Personal/5956/

diff% values are calculated by ((OpenJ9 with OJDK MHs) - Baseline) / Baseline * 100. For Higher is Better? == TRUE positive diff% values suggest better performance for OpenJ9 with OJDK MHs and the raw values are TPS (Transactions per second). For Higher is Better? == FALSE negative diff% values suggest better performance for OpenJ9 with OJDK MHs and the raw values are time in seconds.

JDK8	OpenJ9	OpenJ9 (OJDK MHs)	diff%	Higher == Better?
Startup	3554.76	3615.738	1.72	FALSE
Throughput	4758.766	4735.5	-0.49	TRUE
Throughput (warm)	4828.133	4701.583	-2.62	TRUE

JDK11	OpenJ9	OpenJ9 (OJDK MHs)	diff%	Higher == Better?
Startup	3824.976	3905.571	2.11	FALSE
Throughput	4764.75	4781.333	0.35	TRUE
Throughput (warm)	4810.017	4788.133	-0.45	TRUE

vijaysun-omr · 2024-01-10T20:21:27Z

I think we will need confirmation from (say) Marius Pirvu on the exact throughput situation here for Daytrader7. For example, I would certainly be reluctant to accept 2.62% throughout loss in Daytrader7 warm SCC runs in Java8 on x86 (it is not easy to get that kind of improvement in these kinds of apps being the reason). It is very possible that there is some fluctuation in the score and in fact, there isn't any substantial regression. So we just need to be more certain and debug to reduce the impact if the regression amount is in fact confirmed.

fyi @0xdaryl and @hzongaro if you have thoughts.

singh264 · 2024-07-09T19:27:34Z

@babsingh how can I know the status of the issue as it seems like it is a pre-requisite for #18400 and it is important to document the thoughts?

babsingh added project:MH Used to track Method Handles related work comp:vm labels May 19, 2021

babsingh added the comp:jit label May 20, 2021

babsingh mentioned this issue Jun 6, 2024

Apache Lucene CI builds sometimes fail with OpenJ9 specific issues #18400

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OpenJ9 Perf Results with OJDK MHs enabled #12728

OpenJ9 Perf Results with OJDK MHs enabled #12728

babsingh commented May 19, 2021

nbhuiyan commented May 20, 2021

babsingh commented May 20, 2021 •

edited

Loading

babsingh commented May 20, 2021

0xdaryl commented May 20, 2021 •

edited

Loading

ThanHenderson commented Nov 2, 2023 •

edited

Loading

babsingh commented Nov 3, 2023

ThanHenderson commented Nov 4, 2023 •

edited

Loading

ThanHenderson commented Dec 11, 2023 •

edited

Loading

ThanHenderson commented Dec 12, 2023 •

edited

Loading

vijaysun-omr commented Jan 10, 2024

singh264 commented Jul 9, 2024 •

edited

Loading

OpenJ9 Perf Results with OJDK MHs enabled #12728

OpenJ9 Perf Results with OJDK MHs enabled #12728

Comments

babsingh commented May 19, 2021

Target benchmarks

nbhuiyan commented May 20, 2021

Jruby

red-black (lower=better)

hash (higher = better)

mandelbrot (lower=better)

kwargs (lower=better)

weird_sort (higher=better)

Nashorn (lower=better)

babsingh commented May 20, 2021 • edited Loading

babsingh commented May 20, 2021

0xdaryl commented May 20, 2021 • edited Loading

ThanHenderson commented Nov 2, 2023 • edited Loading

Nashorn JDK11

Setup

Results

babsingh commented Nov 3, 2023

ThanHenderson commented Nov 4, 2023 • edited Loading

JRuby JDK11

Setup

ThanHenderson commented Dec 11, 2023 • edited Loading

Evaluation after OJDK MH backport to JDK 11 and 8

Setup

JDK8

JDK11

JDK17

JRuby

JRuby: JDK8

JRuby: JDK11

JRuby: JDK17

Nashorn

Nashorn: JDK8

Nashorn: JDK11

Nashorn: JDK17

ThanHenderson commented Dec 12, 2023 • edited Loading

DayTrader7

Setup

JDK8

JDK11

vijaysun-omr commented Jan 10, 2024

singh264 commented Jul 9, 2024 • edited Loading

babsingh commented May 20, 2021 •

edited

Loading

0xdaryl commented May 20, 2021 •

edited

Loading

ThanHenderson commented Nov 2, 2023 •

edited

Loading

ThanHenderson commented Nov 4, 2023 •

edited

Loading

ThanHenderson commented Dec 11, 2023 •

edited

Loading

ThanHenderson commented Dec 12, 2023 •

edited

Loading

singh264 commented Jul 9, 2024 •

edited

Loading