Ability to add arbitrary objects to the testing context, for debugging purposes

[vemv]

Clear and concise description of the problem

Many testing frameworks have a facility for adding arbitrary objects (or string representations thereof) to the current context, so that when the given test fails, those objects are printed, helping the programmer understand what went wrong.

Some usual examples:

• In an e2e test, I may want to add the entire http request to the testing context
• I may want to add entire application state to the context: database, caches, etc

Suggested solution

Introduce a new helper that allows the programmer to add objects to the current context.

Iff a given test fails, such extra context, if any, is printed.

Sample usage:

it("blah", async () => {
  const response = await api.post("/foo");
+ context(response, async () => { // add `response` to the testing context, for debugging purposes
    expect(response.status).toBe(200);
  });
});

Alternative

No response

Additional context

No response

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

[hi-ogawa]

Many testing frameworks have a facility for [...]

Can you provide some links to the examples of other frameworks? It would be useful to get the inspiration from them.

[vemv]

Thanks for the response!

I'm most used to clojure.test's testing.

It's string-based, so whatever object one provides is converted to string. That string is always printed in failing tests, with no extra config needed.

Ruby RSpec's RSpec.current_example.metadata is truly object-based and not only can help debugging, but arbitrary other purposes - it can be said to be transient storage. There are misc examples here.

Cheers - V

[hi-ogawa]

Can you also show the example usage (EDIT: ok, this one is on grep) and also how error output looks like with the context? Just having a screenshot would be nice.

[vemv]

Ok!

Test code:

(deftest foo-test
  (testing "Foo behaves correctly"
    (let [foo (/ 42.0 (rand-int 10))]
      (testing (str "Foo is: " (pr-str {:foo foo}))
        (is (neg? foo)
            "Foo is a negative number")))))

Output:

$ lein test :only example.unit-test/foo-test

FAIL in (foo-test) (unit_test.clj:32)
Foo behaves correctly
+ Foo is: {:foo 8.4}
Foo is a negative number
expected: (neg? foo)
  actual: (not (neg? 8.4))

Ran 1 tests containing 1 assertions.
1 failures, 0 errors.

The key thing to highlight is that we can mix and match static contexts (a fixed string known at compile-time: "Foo behaves correctly") with dynamic contexts (values only known at runtime: a random number produced by (rand-int 10), or a http response, state of a database, etc).

Then the test runner concatenates those strings in some or other way (e.g. newline-delimited).

[hi-ogawa]

This is somewhat possible in user land.

https://stackblitz.com/edit/vitest-dev-vitest-qngncqwe?file=test%2Fbasic.test.ts

// this can be defined in a separate `setupFiles`
afterEach((ctx) => {
  const error = ctx.task.result?.errors?.[0];
  if (error) {
    // append extra to the current error message
    error.message += '\nCONTEXT:' + JSON.stringify(ctx.task.meta.someContext);
  }
});

test('x', (ctx) => {
  // setup custom `meta` on `ctx.task` or `getCurrentTest` global
  (getCurrentTest() as any).meta.someContext = {
    hello: 'sqaured(2)',
  };

  expect(squared(2)).toBe(5);
});

I don't think I've seen such feature in js ecosystem, but maybe this is close to playwright's runtime annotation https://playwright.dev/docs/test-annotations#runtime-annotations