Error Model Loading / extract-and-run

[Samoed]

When doing commits, it appears

System.IO.IOException: No space left on device : '/home/runner/runners/2.321.0/_diag/Worker_20250110-152956-utc.log' ...

example run https://github.com/embeddings-benchmark/mteb/actions/runs/12712643686/job/35438686654?pr=1749
@isaac-chung

[isaac-chung]

I have a rough idea what's going on: the base branch in the PR is not main, and the actual one was not passed into extract_model_names.py.

[isaac-chung]

Feel free to skip it while I work on a fix.

[isaac-chung]

There's an error like this one as well:

python scripts/extract_model_names.py
Traceback (most recent call last):
File "/home/runner/work/mteb/mteb/scripts/extract_model_names.py", line 60, in
model_names = extract_model_names(changed_files)
File "/home/runner/work/mteb/mteb/scripts/extract_model_names.py", line 40, in extract_model_names
model_name = next(
File "/home/runner/work/mteb/mteb/scripts/extract_model_names.py", line 42, in
kw.value.value
AttributeError: 'Name' object has no attribute 'value'
make: *** [Makefile:44: model-load-test] Error 1

[Samoed]

What can cause this error?

[isaac-chung]

Problem seems to persist in #1805

[Samoed]

Maybe we should check in tests if json file is correct without downloading models?

[isaac-chung]

Problem seems to persist in #1805

The branch in #1805 indeed has the previous fix merged in. I will take a look at why the files on disk aren't being removed.

Maybe we should check in tests if json file is correct without downloading models?

@Samoed How would that work?

[Samoed]

The script will scan all model metas and check if each model is present in the JSON. If a model is not found, the test will fail. This way, the test won't download any models.

[isaac-chung]

I get the benefit of not downloading any model files, but which JSON file is being checked against? How does this test actually validate that the model loading?

[edit] Oh, by json file, do you mean model_load_failures.json? Hmm that could work. This way, any added model meta will need to be run by the user. In that case, I can add better instructions to help with that.

The core issue of scan_cache_dir failing remains unresolved though, and still might affect the user running the test locally.

[Samoed]

The main issue is figuring out how to add new models to model_load_failures.json. The only approach I can think of is to run the test in two modes: one that downloads the model and another that just performs a json check. I don’t think scan_cache_dir is a big problem since users typically add only 1-2 models at a time.

[isaac-chung]

Good call, let's go with your suggestion. I feel @x-tabdeveloping 's case (adding many models) is rather rare. I'll work on that today and will update when ready 💪

[Samoed]

Maybe this could be handled similarly to tasks' descriptive statistics. The wrapper could write all the information about model downloading to a file, and ModelMeta would only read the data from that file.

[isaac-chung]

The suggestion above works well with adding models, but not updating models. The json check alone does not influence the test outcomes. What we missed was whether the current target key gives the value "None" (means the model loaded without issues, if the script was run).

Here are the cases:

Case	Changes in File Detected	Model Names Exist in JSON	JSON Target Key Gives "None"	Test Outcome
Add New Models	Yes	No	No	❌
Add New Models	Yes	Yes	No	❌
Add New Models	Yes	Yes	Yes	✅
Update Models	Yes	Yes	No	❌
Update Models	Yes	Yes	Yes	✅

In the update models case, if a model passed before and passes again after file changes, git will not detect a change for the json file.

Note

That said, this version is not able to check whether model load has actually been run or not. This means that it does not stop someone from hardcoding MODEL_NAME: "None" into the json file.

[Samoed]

Maybe the model could be run on some mock tasks (or all mock tasks) with the main score and MTEB version added to json. This could be used to check if anything has been updated, and the comparison could be done using a git diff by tag. But this not aligning with previous idea

[isaac-chung]

That could work, if it could stop someone from hardcoding in the json file. I don't think it does :(

Another approach could be only run one model from a changed model file. Right now it runs all models from a changed model file. This might help side-step our issue with scan_cache_dir with minimal updates to the current test setup.