[v2] Refactor evaluators and Abstasks

[Samoed]

I've made some refactoring for the task evaluators, except for Retrieval, which still requires a significant overhaul.

Additionally, I have some suggestions for classification:

• Should we limit it to a single scoring method to avoid reproduction issues, since scoring methods are currently passed in the constructor?
• Should we hardcode split names to avoid reproduction issues, as they are also passed in the constructor?
• Should n_experiments and k become part of the task definitions without requiring changes, to avoid reproduction issues since these are currently passed?

Checklist

• Run tests locally to make sure nothing is broken using make test.
• Run the formatter to format the code using make lint.

[KennethEnevoldsen]

Great additions!

Should we hardcode split names to avoid reproduction issues, as they are also passed in the constructor?

Yes

Should we limit it to a single scoring method to avoid reproduction issues, since scoring methods are currently passed in the constructor?

Hmm not sure what you refer to here

[KennethEnevoldsen]

we could probably just reupload it and remove this part then

[Samoed]

Moved this function to ArXivHierarchicalClusteringP2P.v2, because ArXivHierarchicalClusteringP2P uses same dataset

[KennethEnevoldsen]

you are probably already thinking this, but let us just re-upload it

[KennethEnevoldsen]

is test_cache a Path | None?

[Samoed]

No, this is embeddings of test split

mteb/mteb/evaluation/evaluators/ClassificationEvaluator.py

Lines 145 to 153 in 8d033f3

	if test_cache is None:
	X_test = model.encode(
	self.sentences_test,
	task_name=self.task_name,
	**self.encode_kwargs,
	)
	test_cache = X_test
	else:
	X_test = test_cache

For each experiment there we sample N examples per label for train and use full split for testing

[Samoed]

Should we limit it to a single scoring method to avoid reproduction issues, since scoring methods are currently passed in the constructor?

For now in classification can be used KNN, PytorchKNN and LogReg (default) and I think we can leave only LogReg

[KennethEnevoldsen]

I think we can leave only LogReg

Agree - Set it as a class attribute, so tasks can redefine it if needed (similar to evaluator for summeval)

Then people can just redefine the class if needed.

[Samoed]

I’ve made AbsTaskClassification the parent class for AbsTaskMultilabelClassification because they share the evaluate and _calculate_metrics_from_split functions. I considered making AbsTaskClustering the parent class for AbsTaskClusteringFast, but since they only share _calculate_metrics_from_split, I don’t think it’s as important.