Add Outlines

[yvan-sraka]

Fix #1346, WDYT about the integration with outlines.prompt? (Here, the template argument is just a function that returns a str, so it could be either an @outlines.prompt-decorated function or a regular one)

[rlouf]

Fix #1346, WDYT about the integration with outlines.prompt?

We should allow functions that are not decorated, such as

def build_prompt(a: int) -> str:
    return f"What is {a} squared?"

Because we do not want to force the use of templates on users.

[rlouf]

Also you can deprecate the code in function.py as this is going to replace it.

[rlouf]

This should use either outlines.generate.json, outlines.generate.regex, etc. The interface is necessarily going to be brittle for now. You can take a look at the v1 branch to see how this is going to be more robust with the undergoing refactor.

[yvan-sraka]

Hmm, okay, but which one of those should I use by default? And how should I let users switch to the other one, by adding a fourth optional argument, e.g. generation_kind, WDYT?

[yvan-sraka]

I just realized that maybe I misunderstood the issue, and this PR should infer the correct regex to match the user's required output type. E.g., if a user asks for an int, I should ask outlines.generate.regex something like ^[+-]?[0-9]+$. IIUC, I'm unsure how to scale this to arbitrary return types. Should we define a list of supported basic return types? Using ast.literal_eval was the most flexible solution I came up with, but I'm not sure how well it handles custom data structures. In such cases (when the return type isn't in the list of straightforward Python builtins), I think we should check that users provide types that implement deserialization from JSON (and then use outlines.generate.json), WDYT?

[rlouf]

Let's only accept Pydantic models / JSON Schema strings for now and use outlines.generate.json. We'll be able to easily generalise after we push the refactor in the v1 branch.

[yvan-sraka]

I've updated the PR (and the usage example in the docstring). Does it still look like what we want?

[yvan-sraka]

Fix #1346, WDYT about the integration with outlines.prompt?

We should allow functions that are not decorated, such as

def build_prompt(a: int) -> str:
    return f"What is {a} squared?"

Because we do not want to force the use of templates on users.

That's already how this PR works, I just wanted to confirm that this was the intended design and not an omission!

Also you can deprecate the code in function.py as this is going to replace it.

Should I handle that in this PR or a separate one? By deprecate, do you mean adding a user-facing warning when it's used, rather than removing it entirely?

[rlouf]

Should I handle that in this PR or a separate one? By deprecate, do you mean adding a user-facing warning when it's used, rather than removing it entirely?

We can do it in this PR. Let's add a user-facing warning, and announce that it will be removed in favour of the Outline interface in v1.0

[rlouf]

Let's instantiate the model with the JSON data, like we do here to keep a consistent interface.

[yvan-sraka]

I got this linter warning:

The method "parse_raw" in class "BaseModel" is deprecated
The parse_raw method is deprecated; if your data is JSON use model_validate_json, otherwise load the data then use model_validate instead.Pylance

[rlouf]

You're right, my bad

[yvan-sraka]

I'm a bit frustrated by this failing test, I can't reproduce it locally, and I'm not sure how to work around it :/

[torymur]

Update the tests to not use deprecated Function?: https://github.com/dottxt-ai/outlines/blob/79100b293f861ee8bd74390527ec25cb6fe3b670/tests/test_function.py

(Nice to see the deprecation working properly though!)

[torymur]

Also, found mention in quickstart docs, needs to be updated too:

outlines/docs/quickstart.md

Lines 186 to 190 in 79100b2

	generate_joke = outlines.Function(
	tell_a_joke,
	Joke,
	"microsoft/Phi-3-mini-4k-instruct"
	)

[yvan-sraka]

Also, found mention in quickstart docs, needs to be updated too:

outlines/docs/quickstart.md

Lines 186 to 190 in 79100b2

	generate_joke = outlines.Function(
	tell_a_joke,
	Joke,
	"microsoft/Phi-3-mini-4k-instruct"
	)

Rendered

[yvan-sraka]

I'm a bit frustrated by this failing test, I can't reproduce it locally, and I'm not sure how to work around it :/

=========================== short test summary info ============================
FAILED tests/test_outline.py::test_outline - TypeError: 'Mock' object is not iterable
!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
= 1 failed, 376 passed, 247 skipped, 1 xfailed, 2 xpassed in 1085.82s (0:18:05) =

[yvan-sraka]

I’m definitely struggling with this issue. Thanks to action-tmate, I was able to get an interactive shell in a GitHub Action job, but I’m not sure what to do next? Maybe I should compare my local and CI pip freeze outputs? I’d really like to avoid this kind of issue in the future, I’ve tried to address it in #1387!

[rlouf]

Can you run pytest with the --pdb flag?

[yvan-sraka]

Can you run pytest with the --pdb flag?

Thanks for the tips! I ran it inside the GitHub Action job and got the following traceback:

outlines/fsm/guide.py:76: in cached_create_states_mapping
    return uncached_create_states_mapping(regex_string, tokenizer, *args, **kwargs)
/opt/hostedtoolcache/Python/3.10.16/x64/lib/python3.10/site-packages/outlines_core/fsm/guide.py:141: in create_states_mapping
    return create_states_mapping_from_fsm(regex_fsm, tokenizer, frozen_tokens)
/opt/hostedtoolcache/Python/3.10.16/x64/lib/python3.10/site-packages/outlines_core/fsm/guide.py:178: in create_states_mapping_from_fsm
    states_to_token_maps, empty_token_ids = create_fsm_index_tokenizer(
/opt/hostedtoolcache/Python/3.10.16/x64/lib/python3.10/site-packages/outlines_core/fsm/regex.py:473: in create_fsm_index_tokenizer
    tokens_to_token_ids, empty_token_ids = reduced_vocabulary(tokenizer)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

tokenizer = <Mock name='mock.tokenizer' id='140081937198256'>

    @lru_cache
    def reduced_vocabulary(
        tokenizer,
    ) -> Tuple[Dict[str, List[int]], Set[int]]:
        """Create a map from decoded vocabulary tokens to lists of equivalent token ids."""
        # TODO FIXME: See if we can get the underlying Rust tokenizers from HF and
        # do all this in Rust
        empty_token_ids = set()
        vocabulary: Dict[str, List[int]] = {}
>       for token, token_idx in tokenizer.vocabulary.items():
E       TypeError: 'Mock' object is not iterable

/opt/hostedtoolcache/Python/3.10.16/x64/lib/python3.10/site-packages/outlines_core/fsm/regex.py:394: TypeError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /opt/hostedtoolcache/Python/3.10.16/x64/lib/python3.10/site-packages/outlines_core/fsm/regex.py(394)reduced_vocabulary()
-> for token, token_idx in tokenizer.vocabulary.items():
(Pdb) p tokenizer
<Mock name='mock.tokenizer' id='140081937198256'>
(Pdb) p dir(tokenizer)
['assert_any_call', 'assert_called', 'assert_called_once', 'assert_called_once_with', 'assert_called_with', 'assert_has_calls', 'assert_not_called', 'attach_mock', 'call_args', 'call_args_list', 'call_
count', 'called', 'configure_mock', 'method_calls', 'mock_add_spec', 'mock_calls', 'reset_mock', 'return_value', 'side_effect', 'vocabulary']
(Pdb) p tokenizer.vocabulary
<Mock name='mock.tokenizer.vocabulary' id='140081937198736'>
(Pdb) p tokenizer.vocabulary.items
<Mock name='mock.tokenizer.vocabulary.items' id='140081937198832'>

This made me realize that from outlines_core.fsm.regex import reduced_vocabulary is only used in tests/generate/test_integration_transformers.py, so the test might fail because of an update in outline_core, rather than an issue with the tests in tests/test_outline.py, WDYT?