Any plans to integrate GTE model natively into transformers

[yaswanth19]

Model description

Any plans to integrate gte model natively into transformers as right now we are using this model with trust_remote_code=True argument

Open source status

• The model implementation is available
• The model weights are available

Provide useful links for the implementation

Model Implementation: https://huggingface.co/Alibaba-NLP/new-impl/blob/main/modeling.py
Model Weights: https://huggingface.co/Alibaba-NLP/gte-base-en-v1.5

[yaswanth19]

@ArthurZucker @Rocketknight1 If we do intend to integrate this model then I can work on creating a draft PR.

[mahimairaja]

Is there a place, I can help you to add the model @yaswanth19 ?

[yaswanth19]

@ArthurZucker A gentle ping.
@tomaarsen Can this also be integrated into sentence-transformers

[Rocketknight1]

This seems popular enough to justify an integration, yes. WDYT @tomaarsen?

[tomaarsen]

@Rocketknight1
I suspect there are 3 popular and promising models built on this architecture:

• https://huggingface.co/Alibaba-NLP/gte-large-en-v1.5
• https://huggingface.co/Alibaba-NLP/gte-base-en-v1.5
• https://huggingface.co/Snowflake/snowflake-arctic-embed-m-v2.0

Beyond that, the authors are now using another implementation on top of Qwen:

• https://huggingface.co/Alibaba-NLP/gte-Qwen2-7B-instruct
• https://huggingface.co/Alibaba-NLP/gte-Qwen2-1.5B-instruct

Some of the mechanisms are similar to ModernBERT (I see unpadding), but some differ as well (xformers). It might require a good bit of effort to get everything to line up with transformers, and I think there's a chance that there will be no more big models based on this architecture.

• Tom Aarsen

[yaswanth19]

@Rocketknight1 Should I start implementing support for this model, or do you think the effort outweighs the potential benefit and keep using these models with trust_remote_code?

[Rocketknight1]

Hi @yaswanth19, given @tomaarsen's comment above, I think it's okay to leave them as trust_remote_code models, especially if future versions of gte already exist on a different architecture.

[tomaarsen]

On this topic, the Alibaba team actually just released superior modes based on the new ModernBERT architecture today:

• https://huggingface.co/Alibaba-NLP/gte-modernbert-base
• https://huggingface.co/Alibaba-NLP/gte-reranker-modernbert-base

I imagine that they might not move forward with their previous architecture, especially considering they mention that the only parameter they changed for these compared to their previous models was the base model.

• Tom Aarsen