Htmlunit tuning #715

rschwietzke · 2024-01-27T21:52:54Z

HtmlUnit tuning regarding speed and memory. Current success, more on the memory side.
Please run the full test program, there might be something hidden. The small one looked ok.

E-commerce retailer website

NEW (JDK 17)

Benchmark                                    Mode  Cnt        Score   Error   Units
HtmlUnitBenchmark.simple                     avgt    2  9639101.018           ns/op
HtmlUnitBenchmark.simple:gc.alloc.rate.norm  avgt    2  7612275.912            B/op

OLD (JDK 17)

Benchmark                                    Mode  Cnt         Score   Error   Units
HtmlUnitBenchmark.simple                     avgt    2  10527569.765           ns/op
HtmlUnitBenchmark.simple:gc.alloc.rate.norm  avgt    2   9213538.844            B/op

XC Homepage

NEW (JDK 17)

Benchmark                                    Mode  Cnt       Score   Error   Units
HtmlUnitBenchmark.simple                     avgt    2  397427.602           ns/op
HtmlUnitBenchmark.simple:gc.alloc.rate.norm  avgt    2  383380.466            B/op

OLD (JDK 17)

Benchmark                                    Mode  Cnt       Score   Error   Units
HtmlUnitBenchmark.simple                     avgt    2  411880.438           ns/op
HtmlUnitBenchmark.simple:gc.alloc.rate.norm  avgt    2  454900.410            B/op

likely shared

…n to the thread that created it, removed.

…nized, but churns too much memory when we need a fail-safe iterator

…d sychronized, but churns too much memory when we need a fail-safe iterator" This reverts commit fb8355b1c0b32a43b12b425f492f20898e8b4b1f.

…r lists should not be insanely large and hence we can remove by iterating

… index based structure later, less memory when in use, based on FashHashMap with futher footprint reduction

…by default, only in case we don't find anything, we go the lowercasing and comparision route

…cially when also shared, is more expensive than a potential lower casing operation which in most case might not be needed at all.

…hen asking contains or get due to the more likely cache misses when jumping into an array of hashes

…p content beyond the original position

rbri · 2024-01-28T08:57:16Z

src/main/java/org/htmlunit/util/StringUtils.java

-    private static final Map<String, String> CamelizeCache_ = new HashMap<>();
-    private static final Map<String, String> RootLowercaseCache_ = new HashMap<>();
+    private static final Map<String, String> CamelizeCache_ = new ConcurrentHashMap<>();
+    private static final Map<String, String> RootLowercaseCache_ = new ConcurrentHashMap<>();


RootLowercaseCache_ seems to be no longer used - does having this case makes no sense?

found the answer in the commit comment

rbri · 2024-01-28T08:57:50Z

src/test/java/org/htmlunit/util/RandomUtils.java

+
+import java.util.Random;
+
+public final class RandomUtils {


please add author

rbri · 2024-01-28T10:04:12Z

https://jenkins.wetator.org/job/htmlunit/view/change-requests/job/PR-715/

the failing tests are my ones - i'm still in the middle of a huge refactoring

rbri · 2024-01-28T10:04:28Z

https://jenkins.wetator.org/job/HtmlUnit%20-%20Library%20Tests/view/change-requests/job/PR-715/

lets see if they also pass

rbri · 2024-01-28T12:11:43Z

@shapiroronny is this something you are also interested in? Will be great to see if all the hard work from @rschwietzke has some impact for your use case

rbri · 2024-01-28T13:54:19Z

https://jenkins.wetator.org/job/HtmlUnit%20-%20Library%20Tests/view/change-requests/job/PR-715/2/

Looks great

shapiroronny · 2024-01-28T14:39:22Z

@shapiroronny is this something you are also interested in? Will be great to see if all the hard work from @rschwietzke has some impact for your use case

Hopefully yes. We're disabling javascript for our case and don't do any navigations and very little DOM mutation so not sure which flows are optimized. If initial parsing is faster, that's huge for us.

In any case, we haven't bumped in a while so we're happy upgrading in steps in case this was planned to go in the next major version with Computed Css fixes.

rbri · 2024-01-28T16:22:20Z

Hopefully yes. We're disabling javascript for our case and don't do any navigations and very little DOM mutation so not sure which flows are optimized. If initial parsing is faster, that's huge for us.

Great; that was my understanding of your use case and of @rschwietzke 's goal.
Will merge that soon and we will see....

rschwietzke and others added 19 commits January 27, 2024 22:42

Globally shared caches have to be thread-safe, if it is static, it is

e1b1090

likely shared

Only keep a hashmap for a namespace if we need it

36ae9a8

ConcurrentHashmap is more efficent than a classic synchronized map

faba021

Synchronization for changing a reference on a clone that is only know…

8dede64

…n to the thread that created it, removed.

The clone might want to have the same synchronized list as the source

a788436

Attempt to use a CopyOnWriteArrayList instead of two lists and sychro…

35f0af4

…nized, but churns too much memory when we need a fail-safe iterator

Revert "Attempt to use a CopyOnWriteArrayList instead of two lists an…

aa1474b

…d sychronized, but churns too much memory when we need a fail-safe iterator" This reverts commit fb8355b1c0b32a43b12b425f492f20898e8b4b1f.

Save references, a dedicated lock and a second holder object, listene…

d925d17

…r lists should not be insanely large and hence we can remove by iterating

Different concept for an ordered map that avoid the need for an extra…

c849ab4

… index based structure later, less memory when in use, based on FashHashMap with futher footprint reduction

JavaScript mimetype lookup via FashHashMap and upper and lowercasing …

ff32528

…by default, only in case we don't find anything, we go the lowercasing and comparision route

Removed lowercase caching. Memory access for a large random map, espe…

63984e3

…cially when also shared, is more expensive than a potential lower casing operation which in most case might not be needed at all.

Just run a cheap check first before comparing stuff

2ef72c1

Using OrderHashMap instead of LinkedHashMap plus List

5978331

List should be cheaper than a map for the expected data amount even w…

ed6f4c7

…hen asking contains or get due to the more likely cache misses when jumping into an array of hashes

Lookup of elements is more efficient now

8dc6909

Added performance idea notepad

d1e7ccd

Fixed merge issue

70454f0

Fixing ordered map growth and assuring it is not stuck when looking u…

84d4c24

…p content beyond the original position

Styleguide updates

9208d7b

rbri reviewed Jan 28, 2024

View reviewed changes

src/test/java/org/htmlunit/util/RandomUtils.java

import java.util.Random;

public final class RandomUtils {

Copy link

Member

rbri Jan 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add author

rbri merged commit 9db455d into HtmlUnit:master Jan 28, 2024
2 of 3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Htmlunit tuning #715

Htmlunit tuning #715

rschwietzke commented Jan 27, 2024

rbri Jan 28, 2024

rbri Jan 28, 2024

rbri Jan 28, 2024

rbri commented Jan 28, 2024

rbri commented Jan 28, 2024

rbri commented Jan 28, 2024

rbri commented Jan 28, 2024

shapiroronny commented Jan 28, 2024

rbri commented Jan 28, 2024

Htmlunit tuning #715

Htmlunit tuning #715

Conversation

rschwietzke commented Jan 27, 2024

E-commerce retailer website

NEW (JDK 17)

OLD (JDK 17)

XC Homepage

NEW (JDK 17)

OLD (JDK 17)

rbri Jan 28, 2024

Choose a reason for hiding this comment

rbri Jan 28, 2024

Choose a reason for hiding this comment

rbri Jan 28, 2024

Choose a reason for hiding this comment

rbri commented Jan 28, 2024

rbri commented Jan 28, 2024

rbri commented Jan 28, 2024

rbri commented Jan 28, 2024

shapiroronny commented Jan 28, 2024

rbri commented Jan 28, 2024