Support console output - Switch to Kaocha?

[bobbicodes]

I was looking into what it would take to support printing console output, since this is a common thing people try to do and currently makes the whole test run error without any meaningful message.

While searching for how to capture stdout during a test run, I stumbled upon this: https://github.com/lambdaisland/kaocha/blob/main/doc/plugins/capture_output.md

In other words, it supports that out of the box. And from the looks of it, I think I would rather just use Kaocha than figure out how to do that with our current setup.

I had previously looked into Kaocha when I was choosing a testing framework, and it has another compelling advantage: It has an option to not randomize the testing order, which is exactly what we need!

So why didn't I decide to use it? Because it ran on the JVM, so it was worth avoiding it to gain the quick startup time of Babashka.

This had the unfortunate side effect of me having to actually parse the test files in order to determine the order they are defined, simply to defeat the behavior of clojure.test randomizing the test runs. It's kind of ridiculous, actually, but I couldn't figure out any other way to do it.

But now, Kaocha is compatible with Babashka, so it ticks all of the boxes.

[bobbicodes]

Hmm I'm playing around with it and I can't seem to figure out how to get it to do what I want...

First of all, it only outputs stdout and stderr when a test fails, because it doesn't assume you would need it for a passing test. Sure, I could dig in and figure out how to patch this behavior... but there goes our nice, "out of the box" solution.

Furthermore, there isn't an obvious way to output structured test data to a file... well, there's a junit-xml plugin, but I actually think I'd rather just capture and parse the text output than deal with that.

If I decide not to use Kaocha, I could possibly just overload the various print functions to have them append to a buffer which will then be put into the output field in the json output.

There are more possible print functions than one might think. Right off the top of my head, there is:

• println
• println-str
• pr
• pr-str
• prn
• prn-str

They all behave slightly differently, like the ones with n output a newline, and the str ones um, I think they have different escaping behavior. But whatever... as long as it outputs something it will be an improvement.

[bobbicodes]

Blah... I'm having trouble making that work too. I can't figure out how to make it not use the print functions from clojure.core without doing something really hacky like trying to inject code into the source file. This is starting to get annoying.

To do this correctly, it might be helpful to check out how Kaocha does it: https://github.com/lambdaisland/kaocha/blob/main/src/kaocha/plugin/capture_output.cljc

Apparently this functionality was taken in turn from a library called eftest. I'm curious now so I think I'll go take a look...

[bobbicodes]

The capture-output code from eftest is fairly straightforward. More complicated than I would have thought, but it's only 59 lines, and might be able to just drop into our library.

I don't quite understand where the output is captured... but it shouldn't be too hard to figure out.

In a way this would be the ideal solution, because it would be the least impact way to go. Our test runner works, it's fine, and nothing would reasonably stop it from working, so it might be best to just leave it alone while addressing the console output problem.

Note that eftest also has the same behavior where it only prints the output if the test fails.