doctest cannot match on exceptions with blank lines in them

[Munksgaard]

Elixir and Erlang/OTP versions

Erlang/OTP 27 [erts-15.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns]

Elixir 1.18.1 (compiled with Erlang/OTP 27)

Operating system

NixOS

Current behavior

I have a function that raises the following error under some expected circumstances:

iex> foo(%MyFoo{...})
** (KeyError) key :bar not found in: nil

If you are using the dot syntax, such as map.field, make sure the left-hand side of the dot is a map

I'm tring to introduce a doctest that verifies this behavior, but it seems like that's not possible?

Here is my doctest:

  @doc ~S"""
  ## Examples

      iex> MyFoo.foo(%MyFoo{...})
      ** (KeyError) key :bar not found in: nil

      If you are using the dot syntax, such as map.field, make sure the left-hand side of the dot is a map
  """
  def  foo(%MyFoo{} = my_foo) do
    ...
  end

Upon running the test I get this error:

     Doctest failed: wrong message for KeyError
     expected:
       "key :bar not found in: nil"
     actual:
       "key :bar not found in: nil\n\nIf you are using the dot syntax, such as map.field, make sure the left-hand side of the dot is a map"

Expected behavior

The current behavior follows the documentation, but it's pretty inconvenient. I would expect ExUnit to cut off anything in the error message after a blank line. That would enable me to write the doctest like this:

  @doc ~S"""
  ## Examples

      iex> MyFoo.foo(%MyFoo{...})
      ** (KeyError) key :bar not found in: nil
  """
  def  foo(%MyFoo{} = my_foo) do
    ...
  end

[josevalim]

Correct. The problem is that we don't know exactly how to address this without introducing ambiguities or without requiring a full blown Markdown parsing engine.

One option we considered is to make exceptions match on the message prefix. Because in those case, you most likely don't want to match on the whole message anyway, as it could even lead to fragility. For example, we may rephrase something and now your suite fails while behaviour is effectively the same. In those case, the exception type and the first sentence is enough.

[Munksgaard]

I think that would also work, at least for my use case. Do you have any thoughts about prefix matching vs explicitly only matching until the first blank newline?

[josevalim]

It may be that prefix matching will lead to easier to understand error messages in case they don't match, because we would simply wrap String.starts_with?. The other one may require additional functions. I'd have to play with it.

[Munksgaard]

Good point. I'm easy either way.

[sabiwara]

A nice way to address this could be to add some support for ellipsis in doctests, i.e. be able to add ... where you want to ignore some parts of the error message (because too verbose or even in some cases, non-deterministic parts e.g. it's inspecting a pid):

    iex> MyFoo.foo(%MyFoo{...})
    ** (KeyError) key :bar not found in: nil
    ...

    iex> MyFoo.foo(self())
    ** (MyError) something happened with pid: ...

Prior-art wise, I think that Python had something like this: https://docs.python.org/3/library/doctest.html#doctest.ELLIPSIS.

Of course, there might be a lot of edge cases to consider and it's probably not so trivial to add.

[josevalim]

I think this would actually be quite trivial to implement, both at compile-time and runtime, it would be clearer and backwards compatible, so it gets a 👍 from me.

[sabiwara]

Will try to send a PR later this week then!