Skip to content

Issues: embeddings-benchmark/mteb

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

130 Open 493 Closed
130 Open 493 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Benchmark tasks are incorrectly selected
#1757 opened Jan 10, 2025 by Samoed
Error Model Loading / extract-and-run
#1756 opened Jan 10, 2025 by Samoed
Leaderboard: Share this benchmark does not work for custom benchmarks
#1755 opened Jan 10, 2025 by Muennighoff
Leaderboard: SFR-Embedding results don't match between old and new
#1754 opened Jan 10, 2025 by Muennighoff
Leaderboard: stella results don't match between new and old leaderboard
#1753 opened Jan 10, 2025 by Muennighoff
2
Leaderboard: K instead of M
#1752 opened Jan 10, 2025 by Muennighoff
Leaderboard: Speed up get_scores()
#1740 opened Jan 9, 2025 by x-tabdeveloping
2
downsample MIRACLReranking enhancement New feature or request good first issue Good for newcomers
#1738 opened Jan 9, 2025 by KennethEnevoldsen
Downsample MIRACLReranking using MIRACLRetrievalHardNegatives enhancement New feature or request good first issue Good for newcomers
#1737 opened Jan 9, 2025 by KennethEnevoldsen
Combine different version of the Encoder interface [v2]
#1735 opened Jan 8, 2025 by KennethEnevoldsen
Potentially relevant multi turn rag dataset new dataset Issues related to adding a new task or dataset
#1732 opened Jan 8, 2025 by KennethEnevoldsen
Non Cosine-Sim Similarity functions for Sentence Transformer models are broken in v2.0.0
#1731 opened Jan 8, 2025 by orionw
1
5
Remove "memory_usage" from ModelMeta
#1729 opened Jan 8, 2025 by KennethEnevoldsen
How should we handle gated models when testing
#1727 opened Jan 8, 2025 by KennethEnevoldsen
2
Problem loading sentence-transformer/multi-qa-MiniLM-L6-cos-v1
#1726 opened Jan 8, 2025 by KennethEnevoldsen
Add support for STS evaluation for Cross-Encoders
#1725 opened Jan 8, 2025 by sam-hey
How to calculate MTEB score from individual files?
#1721 opened Jan 7, 2025 by orionw
2
Leaderboard: Add training data annotations for models good first issue Good for newcomers leaderboard issues related to the leaderboard
#1720 opened Jan 7, 2025 by x-tabdeveloping
1
Inconsistent casing for MIRACL metrics
#1718 opened Jan 6, 2025 by Muennighoff
1
malenia1/ternary-weight-embedding does not load weights
#1717 opened Jan 6, 2025 by Muennighoff
Add test to ensure that the leaderboard can build leaderboard issues related to the leaderboard
#1715 opened Jan 6, 2025 by KennethEnevoldsen
There are multiple 'mteb/arguana' configurations in the cache: default, corpus, queries with HF_HUB_OFFLINE=1
#1714 opened Jan 6, 2025 by Bhavya6187
3
integrate VSI-BENCH modality: image The image extension of MTEB new benchmark Issues related to adding a new benchmark
#1712 opened Jan 6, 2025 by Samoed
Add MMEB modality: image The image extension of MTEB new benchmark Issues related to adding a new benchmark
#1709 opened Jan 5, 2025 by Muennighoff
May I ask if MTEB supports evaluating downloaded data？
#1701 opened Jan 4, 2025 by BtlWolf
4
ProTip! Find all open issues with in progress development work with linked:pr.