Malformed config when saving & loading locally custom models

[Alicimo]

System Info

• transformers version: 4.47.1
• Platform: Linux-6.8.0-51-generic-x86_64-with-glibc2.39
• Python version: 3.11.10
• Huggingface_hub version: 0.27.0
• Safetensors version: 0.4.5
• Accelerate version: 1.2.1
• Accelerate config: not found
• PyTorch version (GPU?): 2.5.1+cu124 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using distributed or parallel set-up in script?: NA
• Using GPU in script?: Yes, but also NA
• GPU type: NVIDIA GeForce RTX 4090

Who can help?

@ArthurZucker

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import AutoModel, AutoTokenizer, AutoConfig

# Load a custom model from the Hub
model = AutoModel.from_pretrained("jinaai/jina-embeddings-v2-base-de", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("jinaai/jina-embeddings-v2-base-de")

# Save them locally
model.save_pretrained("tmp")
tokenizer.save_pretrained("tmp")

# Now load that model again, via a Config
config = AutoConfig.from_pretrained("tmp", trust_remote_code=True)
print(config.__class__)
# <class 'transformers_modules.jinaai.jina-bert-implementation.f3ec4cf7de7e561007f27c9efc7148b0bd713f81.configuration_bert.JinaBertConfig'>
local_model = AutoModel.from_pretrained("tmp", config=config, trust_remote_code=True)

# And save it again
local_model.save_pretrained("tmp_2")
tokenizer.save_pretrained("tmp_2")

# Now load that model again, via a Config
config_2 = AutoConfig.from_pretrained("tmp_2", trust_remote_code=True)
print(config_2.__class__)
# <class 'transformers_modules.tmp_2.configuration_bert.JinaBertConfig'>
# The second time around, the config is not the same as the first time
local_model_2 = AutoModel.from_pretrained("tmp_2", config=config_2, trust_remote_code=True)
# ValueError: The model class you are passing has a `config_class` attribute that is not consistent
# with the config class you passed (model has
# <class 'transformers_modules.jinaai.jina-bert-implementation.f3ec4cf7de7e561007f27c9efc7148b0bd713f81.configuration_bert.JinaBertConfig'>
# and you passed <class 'transformers_modules.tmp_2.configuration_bert.JinaBertConfig'>. Fix one of those so they match!

Expected behavior

Model should load without raising a ValueError

[Rocketknight1]

This is related to my PR #29854, and it's definitely a bug, yes. Let me see if I can loosen up that test a little.