Improve MethodHandle direct dispatch J2I-prevention transformations

[jdmpapin]

...by avoiding breaking the block containing the original call. There isn't much opportunity to optimize the inline logic anyway, so it's better to keep the IL neater.

On code generators that support doing so (initially just x86-64), calls to linkToStatic, linkToSpecial, and invokeBasic that have not been refined are now changed in-place into calls to a new JIT non-helper <jitDispatchJ9Method>, which takes the callee J9Method as an additional (first) argument. It has JIT private linkage, but the extra argument is treated specially so that the other arguments are passed in exactly the way expected by (any JIT-compiled body for) the actual callee. The code generator generates a sequence that determines whether the callee is compiled and jumps to the JIT entry point if it is. Otherwise, it jumps out of line to do a J2I transition.

If the environment variable TR_stressJitDispatchJ9MethodJ2I is set, then the back end generates an unconditional jump to the J2I path, exercising it even when the callee is compiled.

---

There is one additional commit to get rid of the mov edi, <imm> instruction in the call snippet when calling an OSR induction helper on x86-32, mostly for consistency.

---

This depends on eclipse-omr/omr#7090

[jdmpapin]

Updated to undo the change to J9_THUNK_MAX_ENCODED_BYTES

[jdmpapin]

Rebased to fix a conflict

[0xdaryl]

@gacholio : could you review the proposed changes in codert_vm please?

[gacholio]

Why is the new assembly helper only on X, when there are clearly changes on other architectures here?

[gacholio]

• Make J2I thunks usable for static methods

I'm not sure this is valid - the stack walker distinguishes virtual and static calls. I'll have to look at the walker and transition code to be sure if this is bad or not.

Is this an attempt to save the generated code for backstoring arguments in static dispatch snippets?

[jdmpapin]

Is this an attempt to save the generated code for backstoring arguments in static dispatch snippets?

I wrote it this way because originally I had moved all of the logic out to assembly, and I had in mind the way we do the direct dispatches from PicBuilder. In that setting it didn't occur to me to jump back from the helper into the JIT body to run a snippet. But then it turned out that it was slow to have the JIT-to-JIT path out of line, so I inlined it

I was writing a long reply answering about other architectures and detailing why I think the strategy in this PR should work

(click for very abridged version if interested)

• The changes on other platforms were only to adapt to the change in the API of the J2I thunk management functions
• I think this strategy should work because the virtual and static J2I paths in the VM seem to differ only in how they determine the callee J9Method, and because when the JIT generates a direct call to an interpreted method, we already generate the call in the same way for both virtual and static callees, so the stack walker can't expect the compiled code to look different in those two cases

but as I was writing that, I saw how this change could be much less invasive if I use (almost) the same interpreter call snippet we use for direct calls. So I'll try that. Thanks @gacholio for the nudge

[jdmpapin]

Updated to leave J2I thunks alone and instead use the call snippet

[0xdaryl]

Jenkins test sanity all jdk11,jdk21 depends eclipse-omr/omr#7090

[0xdaryl]

Jenkins test sanity all jdk11,jdk21 depends eclipse-omr/omr#7090

Previous CI was a resounding failure. Restarting.

[0xdaryl]

Jenkins test sanity.functional,sanity.openjdk all jdk8,jdk11,jdk17,jdk21 depends eclipse-omr/omr#7090

[0xdaryl]

Jenkins test sanity.functional,sanity.openjdk all jdk8,jdk17 depends eclipse-omr/omr#7090

[pshipton]

I killed that PR build, it's likely to take out jenkins. Pls avoid testing the unb platforms: amac, xlinux, xmac. I've disabled all the unb plinux so that platform should be ok, although we have limited machines at osu.

[0xdaryl]

Jenkins test sanity.functional win,alinux,plinux,zlinux jdk17 depends eclipse-omr/omr#7090

[0xdaryl]

I'm satisfied that a sufficient coverage of tests have passed on this and that the effort to manually shepherd the jobs failing for infrastructure reasons is not a productive use of time. That, coupled with Devin's own internal testing on this, leads me to conclude that this is safe to merge. I will do a coordinated merge early in the morning barring a reasonable objection @pshipton @hzongaro @jdmpapin .

[jdmpapin]

I don't believe a coordinated merge is necessary for this change. It should be fine to merge eclipse-omr/omr#7090 first (but please check that you agree)