j9vmem__testOverlappingSegments in cmdLineTester_pltest_0 seems to segfault if unable to allocate any memory at all

[cjjdespres]

I have been running sanity.functional locally (in containers in a fyre VM, anyway) and have been getting a segfault in cmdLineTester_pltest_0. I believe this is because the subtest j9vmem__testOverlappingSegments is unable to allocate any memory at all:

 [ERR] Starting test j9vmem__testOverlappingSegments
 [ERR]   Cycles: 10000
 [ERR] 
 [ERR]   Segment           Start             End               Size              Keep cycle
 [ERR]   Failed to get memory. Error: Cannot allocate memory.
 [ERR]   Ignoring memory allocation failure(0 of 10000 loops finished).
 [ERR]   
 [ERR] =========================
 [ERR]   0 cycles completed
 [ERR]   0 segments freed
 [ERR] 
 [ERR] 
 [ERR] An unhandled error (24) has occurred.
 [ERR] J9Generic_Signal_Number=00000018
 [ERR] Signal_Number=0000000b
 [ERR] Error_Value=00000000
 [ERR] Signal_Code=00000001
 [ERR] Handler1=00005565C7619FE0
 [ERR] Handler2=00007FDFA7B00B40
 [ERR] InaccessibleAddress=0000000000000018
 [ERR] RDI=0000000000000018
 [ERR] RSI=0000000000000001
 [ERR] RAX=00007FDFA7B49740
 [ERR] RBX=0000000000000000
 [ERR] RCX=00007FDFA79ADA7B
 [ERR] RDX=0000000000000000
 [ERR] R8=0000000000000000
 [ERR] R9=0000000000000000
 [ERR] R10=0000000000000000
 [ERR] R11=0000000000000293
 [ERR] R12=0000000000001000
 [ERR] R13=00000000FFFFFFFF
 [ERR] R14=0000000000000000
 [ERR] R15=0000000000000000
 [ERR] RIP=00007FDFA7B27598
 [ERR] RSP=00007FFD81BE1568
 [ERR] EFlags=0000000000010246
 [ERR] RBP=0000000000000000
 [ERR] ERR=0000000000000004
 [ERR] TRAPNO=000000000000000E
 [ERR] OLDMASK=0000000000000000
 [ERR] CR2=0000000000000018
 [ERR] xmm0=0000000000000000
 [ERR] xmm1=0000000000000000
 [ERR] xmm2=0053534552444441
 [ERR] xmm3=415353454d5f434c
 [ERR] xmm4=657461636f6c6c61
 [ERR] xmm5=7272457473616c09
 [ERR] xmm6=7365747962203030
 [ERR] xmm7=6f7420656c62616e
 [ERR] xmm8=2e74657375706300
 [ERR] xmm9=0000000000000000
 [ERR] xmm10=0000001000000010
 [ERR] xmm11=00fe00fe00fe00fe
 [ERR] xmm12=00fe00fe00fe00fe
 [ERR] xmm13=0000000000000000
 [ERR] xmm14=0000000000000000
 [ERR] xmm15=0000000000000000
 [ERR] Module=/root/hostdir/openj9-openjdk-jdk17/build/linux-x86_64-server-release/images/jdk/bin/../lib/default/libj9prt29.so
 [ERR] Module_base_address=00007FDFA7AD7000

The backtrace I get is this:

#0  subtractAtomic (address=0x18, value=1) at /root/hostdir/openj9-openjdk-jdk17/omr/util/omrutil/AtomicFunctions.cpp:65
#1  0x00007fc2a17261e6 in omrmem_categories_decrement_counters (category=category@entry=0x0, size=size@entry=4096) at /root/hostdir/openj9-openjdk-jdk17/omr/port/common/omrmemcategories.c:97
#2  0x00007fc2a173faba in omrvmem_free_memory (portLibrary=<optimized out>, address=<optimized out>, byteAmount=4096, identifier=<optimized out>)
    at /root/hostdir/openj9-openjdk-jdk17/omr/port/linux/omrvmem.c:761
#3  0x0000564605d4ea61 in j9vmem_testOverlappingSegments ()
#4  0x0000564605d4ed0a in j9vmem_runTests ()
#5  0x0000564605d50178 in signalProtectedMain ()
#6  0x00007fc2a172d7f1 in omrsig_protect (portLibrary=0x7ffd3339be70, fn=0x564605d4f6c0 <signalProtectedMain>, fn_arg=0x7ffd3339be40, handler=0x564605d5afe0 <genericSignalHandler>, handler_arg=0x0, 
    flags=504, result=0x7ffd3339be20) at /root/hostdir/openj9-openjdk-jdk17/omr/port/unix/omrsignal.c:425
#7  0x0000564605d22a07 in main ()

(My debug information seems to be messed up a little - sorry).

I think the failure is happening here:

openj9/runtime/tests/port/j9vmemTest.c

Lines 3458 to 3468 in a56cbdf

	/* Free remaining segments */
	freed = 0;
	for (j = 0; j < CYCLES; j++) {
	if (keepCycles[j] >= i) {
	I_32 rc = j9vmem_free_memory(vmemID[j].address, vmemParams[j].byteAmount, &vmemID[j]);
	if (0 == rc) {
	freed++;
	}
	}
	}
	outputComment(PORTLIB, "%d remaining segments freed\n\n", freed);

and what's happening is:

1. We fail in the first cycle without ever allocating anything, then jump to exit (so i == 0).
2. We have that keepCycles[0] >= 0 because keepCycles was memset to 0 earlier in the function
3. We try to free memory that was never allocated, eventually crashing because the category of the memory is NULL.

[cjjdespres]

The problem was actually a little worse - if the test failed in the very first cycle, the cleanup procedure would attempt to deallocate every segment j from 1 to CYCLES - 1, because keepCycles[j] would always be zero. The same erroneous freeing would happen for segment 0 only if the initial memory allocation failed, like I said above.

[cjjdespres]

The test cmdLineTester_pltest_0 still fails in various other ways for me, presumably because I'm running in containers (or I've set up the tests incorrectly), but #19517 does fix this particular issue.