Support the file attribute under <testcase>

[wcedmisten-reify]

The CircleCI docs expect a file attribute under the <testcase> or <testsuite> tags to lookup test timings by filename.

https://circleci.com/docs/use-the-circleci-cli-to-split-tests/#junit-xml-reports

Although I can't find this mentioned anywhere in the Junit schemas linked in the README, would you consider supporting this in your plugin?

[alysbrooks]

Hi @wcedmisten-reify, thanks for suggesting this.

So implementing this would allow clicking on a file name to see all the timing for that file? Am I understanding you (and the CircleCI docs) correctly? That definitely seems useful.

[wcedmisten-reify]

I'm not sure if there are any UI changes related to this feature, but the main benefit is that CircleCI can parallelize the tests based on the total execution time of a file.

For example, suppose there are 3 files containing unit tests, A, B, and C. The combined execution time for the tests in file A take 2 minutes to run, and files B and C take 1 minute each. Let's say the CircleCI config is parallelized with 2 workers, then the optimal distribution is to run A on worker 1 and B and C on worker 2.

The problem is that currently CircleCI knows the timings at a test level, but it isn't able to link those timings back to the file containing each test. So the file attribute is needed to establish that linking.

Hope this helps provide more context!

[lambduhh]

Meowdy @wcedmisten-reify

Thanks for bringing this to our attention! I reviewed during triage and introducing the file attribute in our JUnit XML outputs could significantly benefit CircleCI users. This enhancement not only aims to speed up test runs but becomes high priority since it could also lead to cost savings by improving test parallelization efficiency.

Rationale: CircleCI calculates charges based on actual test run times, not the aggregate job time. Efficient parallelization means less idle time and, consequently, reduced costs for users with extensive test suites. This feature would enable CircleCI to allocate tests based on precise file execution times, optimizing resource use and decreasing overall test duration.

In short, by making this update, we're looking at faster CI builds and potential savings for our users on CircleCI. It’s a strategic improvement that aligns with our commitment to enhance user experience and operational efficiency.

I'm excited about the potential here and believe it deserves our immediate attention. Changing status to "candidate" and updating to "high" priority.

References:
https://circleci.com/docs/parallelism-faster-jobs/

[wcedmisten-reify]

It looks like there is already a flag that addresses this issue: --junit-xml-add-location-metadata

With it, I was able to get CircleCI test splitting working as expected (although I did need a hacky sed command to convert the absolute paths to relative ones in order for CircleCI to recognize the timing data).

Is there a reason this flag isn't enabled by default? I would propose enabling it unless there are serious objections to that behavior

[nnichols]

I updated the linked PR to add a new flag to switch between relative/absolute pathing: --junit-xml-use-relative-path-in-location. (Biasing towards backwards compatibility in case different CI Platforms have different pathing expectations).

I don't have a dog in the race for enabling --junit-xml-add-location-metadata by default or not. I'll happily defer to the maintainers on that one