Reenable OpenAI Tokenizer

[yuhongsun96]

Description

Reenable OpenAI Tokenizer

This is used for

• Processing messages / text content for GPT models
• Embeddings (specifically defined in the EmbeddingModel init)

References:

1. OpenAI Cookbook
2. LiteLLM tokenizing
3. Previous branch
   

How Has This Been Tested?

N/A

Accepted Risk (provide if relevant)

This tokenizer has caused issues before but it is significantly different from the other one we use which leads to problems at embedding time.

Related Issue(s) (provide if relevant)

N/A

Mental Checklist:

• All of the automated tests pass
• All PR comments are addressed and marked resolved
• If there are migrations, they have been rebased to latest main
• If there are new dependencies, they are added to the requirements
• If there are new environment variables, they are added to all of the deployment methods
• If there are new APIs that don't require auth, they are added to PUBLIC_ENDPOINT_SPECS
• Docker images build and basic functionalities work
• Author has done a final read through of the PR right before merge

Backporting (check the box to trigger backport action)

Note: You have to check that the action passes, otherwise resolve the conflicts manually and tag the patches.

• This PR should be backported (make sure to check that the backport attempt succeeds)

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
internal-search	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Nov 8, 2024 10:40pm