Compilation Errrors

[yogendrahexo]

I'm getting compilation errors. All the D4J projects are failing to compile even at the class level because of the wrong format of test generations. The import statements are getting written inside the class or duplicate imports, causing errors. In some cases, class-level compilation succeeds but only with empty tests. But, Method-level compilation is failing in them.

Compile errors: {'illegal start of type': 17, 'cannot find symbol': 8, '<identifier> expected': 6, 'package org.mockito does not exist': 2, 'repeated modifier': 1}

1. ERROR: Chart_15 failed compilation even though it was given an empty test class.

2. Errors related to Mockito:
   You mentioned Mockito 5 in the instructions, but 3.12.4 was the version in the dependency_analyzer script. I tried both versions (5.14.2 and 3.12.4), but I'm still getting Mockito errors in some compilations:
   package org.mockito does not exist
   This error occurred in the Csv project and a few others, but not in all projects.

3. Method-Level Compilation Failures:

   • method removeDomainMarker not found in org/jfree/chart/plot/CategoryPlot
   • method skip not found in org/apache/commons/codec/binary/BaseNCodecInputStream

---

Project Setup, Steps followed:

I have the following files in the frameworks/projects/lib directory:
Mockito-core-3.12.4.jar, mockito-core-5.14.2.jar, mockito-junit-jupiter-3.12.4.jar, junit-jupiter-api-5.7.2.jar, junit-jupiter-params-5.0.0.jar, powermock-api-mockito2-1.7.4.jar, powermock-core-1.7.4.jar, apiguardian-api-1.1.0.jar, powermock-module-junit4-1.7.4.jar, byte-buddy-1.14.11.jar, byte-buddy-agent-1.14.11.jar, objenesis-3.3.jar, hamcrest-2.1.jar

1. Checked out all the D4J repos (buggy and fixed) and placed them in a folder named projectName_id/fixed and projectName_id/buggy.
2. Copied the source_data.jsonl file into the data/prompts/ folder.
3. Copied the d4j2_fixed_info folder into the data directory.
4. Generated prompts using the script rq1/generate_prompts.py and retrieved content from the generated file. I sent this content to the LLM to generate output and added the response in the input data with the completion key. and palced in the data/outputs/gpt4o_comment_extend_full.jsonl
   I'm using GPT-4o right now.

Current Issues:

• The script is correctly extracting methods, imports, and classes.
• But, the test generation process is not working as expected:
  • Import statements are sometimes placed inside the class, disrupting the structure of the test template.
  • Duplicate imports, missing functions or different function names.

[LeonYang95]

Hi, thanks for your issue. Can you share some GPT-4o results so that I can debug with? That would be very helpful. Thank you in advance.

[yogendrahexo]

Thanks for your response. This is a file with gpt output as completion here gpt4o_comment_extend_full2.zip

this jsonl have the gpt outputs for all the bugs of codec and compress projects.

Here are the keys in the provided JSON:

{
  "id": "",
  "strategy": "",
  "format": "",
  "ablation": "",
  "focal_method": "",
  "content": "",
  "class_name": "",
  "method_signature": "",
  "is_public": "",
  "role": "",
  "custom_id": "",
  "completion": ""
}

[jundaz]

"Import statements are sometimes placed inside the class, disrupting the structure of the test template."
I fixed this one, this happens because the need_prefix parameter in parse_fields_from_class_code is default to true, so any code block is wrapped with a class declaration so the import are treated as field, simply change the calling of this function in analyze_outputs function in output_analyzer.py to parse_fields_from_class_code(cur_content, need_prefix=False) will fix this problem.
A more robust way of doing this might be changing the need prefix judgement to using a if statement to detect if a class already exists, but this then involves other consideration about where to put this prefix (since you still might have import statement) when the class declaration is missing and I observe that in pretty much any case the llm generated results already included a class declaration, so set need_prefix = false is probably the easiest way.

[LeonYang95]

@jundaz Thanks for your comment. This may caused by our experimental settings. By default, we applied some strategies to generate better unit tests. For example, we used "extend" to help reduce the compilation errors occurred in the generated unit tests. And in "extend" mode, LLMs' response should be a test classes without class definition (at least we hope it should be, but there is no guarantee). So we added headers(prefixes) like public class ABC { in multiple parsing location since tree-sitter only accepts complete test class string as input. Therefore, some default parameter values, such as the need_prefix is set to True.

@yogendrahexo Could you please try change the value of the strategy parameter from "extend" to "generation". The generation mode is more suitable when the output of LLM is a complete test class.

I am currently working on a more robust version of our benchmark, and will keep you guys posted.

[yogendrahexo]

@LeonYang95 @jundaz I tried both options. I was initially using extend_full, but then I also tried generation_full along with no_class_other_methods, and neither of them worked. I also tried setting need_prefix = False, but that didn't work either.

With need_prefix = False the import statements issue gets fixed to an extent but the dependency errors are still there. Even when i;ve all configured and those dependencies work in one project but not in other.