Add @huggingface/ollama-utils

[ngxson]

With the ollama compatibility layer already been up and running on HF hub for a while now, we want to open-source part of the integration for (1) provide more transparency, and (2) for encouraging the community to contribute.

Therefore, we decided to publish @huggingface/ollama-utils, a package containing tools that power this integration.

For now, the only tool that we provide is chat-template.ts, a tool that takes a parsed GGUF config from @huggingface/gguf as input, then returns the ollama Go template.

chat-template.ts

This module expose one single convertGGUFTemplateToOllama function. It works by trying (by order) these mechanisms:

1. Try to find a matched template already exist on ollama hub. To make this possible, we regularly dump a list of public templates from ollama.com using scripts/generate-automap.ts
2. If no exact match is find, we still check against the list of official templates, but now only match a list of special tokens. These tokens are extracted using regex RE_SPECIAL_TOKEN
3. If we still can't find a match, we search inside CUSTOM_TEMPLATE_MAPPING, a list of hand-picked template mapping
4. If all above still failed, we try parsing jinja template and try converting it to Go template. This works almost all of the time, except if the template is not supported by @huggingface/jinja. See convertJinjaToGoTemplate

[ngxson]

TODO: complete this @ngxson

[coyotte508]

If you feel up to it you can use tshy instead, like in @hugginface/tasks.

It has more accurate exports / better compatibility with all that includes the package (and as long as you don't have different code in browser/node it should be easy to configure)

[coyotte508]

btw the github action to publish should be added (with optionnally a check to check the @huggingface/jinja is up to date, like @huggingface/hub does with @huggingface/tasks)

[julien-c]

neat!

[Vaibhavs10]

don't know enough typescript to review, but two questions/comments from myside:

1. maybe in a follow-up PR we should add a scraper and something to update the mapping?
2. via comments/ README/ docs we should clarify how users can add their own exception as well