diff --git a/dbpedia/LC-QuAD v2.md b/dbpedia/LC-QuAD v2.md index f3ba7bb6..7b7a9192 100644 --- a/dbpedia/LC-QuAD v2.md +++ b/dbpedia/LC-QuAD v2.md @@ -3,33 +3,34 @@ name: LC-QuAD v2 datasetUrl: https://figshare.com/projects/LCQuAD_2_0/62270 --- -| Model / System | Year | Precision | Recall | F1 | Language | Reported by | Gold Entity | -| :-----------------------: | :--: | :-------: | :----: | :---: | :------: | :----------------------------------------------------------------------------------: | :---------: | -| T5-Small | 2022 | - | - | 92 | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ | -| T5-Base | 2022 | - | - | 91 | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ | -| PGN-BERT-BERT | 2022 | - | - | 86 | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ | -| SGPT_Q,K [1] | 2022 | - | - | 89.04 | EN | [Al Hasan Rony et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9815253) | ✅ | -| PGN-BERT | 2022 | - | - | 77 | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ | -| NSpM [2] | 2022 | - | - | 66.47 | EN | [Al Hasan Rony et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9815253) | ✅ | -| BART | 2022 | - | - | 64 | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ | -| Zou et al. + Bert | 2021 | - | - | 59.30 | EN | [Zou et al.](https://arxiv.org/pdf/2111.06086.pdf) | ✅ | -| CLC | 2021 | - | - | 59 | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ | -| Multi-hop QGG | 2020 | - | - | 53 | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ | -| Zou et al. + Tencent Word | 2021 | - | - | 52.90 | EN | [Zou et al.](https://arxiv.org/pdf/2111.06086.pdf) | ✅ | -| Multi-hop QGG | 2021 | - | - | 52.60 | EN | [Zou et al.](https://arxiv.org/pdf/2111.06086.pdf) | ✅ | -| AQG-net | 2021 | - | - | 44.90 | EN | [Zou et al.](https://arxiv.org/pdf/2111.06086.pdf) | ✅ | -| SGPT_Q [3] | 2022 | - | - | 83.45 | EN | [Al Hasan Rony et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9815253) | ❌ | -| ChatGPT | 2023 | - | - | 42.76 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | ❌ | -| GPT-3.5v3 | 2023 | - | - | 39.04 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | ❌ | -| GPT-3.5v2 | 2023 | - | - | 33.77 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | ❌ | -| GPT-3 | 2023 | - | - | 33.04 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | ❌ | -| FLAN-T5 | 2023 | - | - | 30.14 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | ❌ | -| UNIQORN | 2021 | 33.1 | - | - | EN | [Pramanik et al.](https://arxiv.org/abs/2108.08614) | ❌ | -| QAnswer | 2020 | 30.80 | - | - | EN | [Pramanik et al.](https://arxiv.org/abs/2108.08614) | ❌ | -| GraftNet | 2018 | 19.7 | - | - | EN | [Christmann P. et al](https://arxiv.org/pdf/2108.08597.pdf) | ❌ | -| ElNeuQA-ConvS2S [1] | 2021 | 26.90 | 27 | 26.90 | EN | [Diomedi, Hogan](https://arxiv.org/pdf/2107.02865.pdf) | ❌ | -| GRAFT-Net + Clocq [2] | 2022 | 19.70 | - | - | EN | [Christmann P. et al](https://arxiv.org/pdf/2108.08597.pdf) | ❌ | -| Platypus | 2018 | 3.6 | - | - | EN | [Pramanik et al.](https://arxiv.org/abs/2108.08614) | ❌ | -| Pullnet | 2019 | 1.1 | - | - | EN | [Pramanik et al.](https://arxiv.org/abs/2108.08614) | ❌ | -| UNIK-QA | 2020 | 0.5 | - | - | EN | [Pramanik et al.](https://arxiv.org/abs/2108.08614) | ❌ | -| GETT-QA [4] | 2023 | 40.3 | - | - | EN | [Banerjee et al.](https://arxiv.org/pdf/2303.13284.pdf) | ❌ | +| Model / System | Year | Precision | Recall | F1 | Hits@1 | Language | Reported by | Gold Entity | +| :-----------------------: | :--: | :-------: | :----: | :---: | :----: | :------: | :----------------------------------------------------------------------------------: | :---------: | +| T5-Small | 2022 | - | - | 92 | - | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ | +| T5-Base | 2022 | - | - | 91 | - | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ | +| PGN-BERT-BERT | 2022 | - | - | 86 | - | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ | +| SGPT_Q,K [1] | 2022 | - | - | 89.04 | - | EN | [Al Hasan Rony et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9815253) | ✅ | +| PGN-BERT | 2022 | - | - | 77 | - | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ | +| NSpM [2] | 2022 | - | - | 66.47 | - | EN | [Al Hasan Rony et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9815253) | ✅ | +| BART | 2022 | - | - | 64 | - | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ | +| Zou et al. + Bert | 2021 | - | - | 59.30 | - | EN | [Zou et al.](https://arxiv.org/pdf/2111.06086.pdf) | ✅ | +| CLC | 2021 | - | - | 59 | - | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ | +| Multi-hop QGG | 2020 | - | - | 53 | - | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ | +| Zou et al. + Tencent Word | 2021 | - | - | 52.90 | - | EN | [Zou et al.](https://arxiv.org/pdf/2111.06086.pdf) | ✅ | +| Multi-hop QGG | 2021 | - | - | 52.60 | - | EN | [Zou et al.](https://arxiv.org/pdf/2111.06086.pdf) | ✅ | +| AQG-net | 2021 | - | - | 44.90 | - | EN | [Zou et al.](https://arxiv.org/pdf/2111.06086.pdf) | ✅ | +| SGPT_Q [3] | 2022 | - | - | 83.45 | - | EN | [Al Hasan Rony et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9815253) | ❌ | +| W. Han et al. | 2023 | - | - | - | 42.6 | EN | [Han et al.](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_39) | ❌ | +| ChatGPT | 2023 | - | - | 42.76 | - | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | ❌ | +| GPT-3.5v3 | 2023 | - | - | 39.04 | - | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | ❌ | +| GPT-3.5v2 | 2023 | - | - | 33.77 | - | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | ❌ | +| GPT-3 | 2023 | - | - | 33.04 | - | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | ❌ | +| FLAN-T5 | 2023 | - | - | 30.14 | - | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | ❌ | +| UNIQORN | 2021 | 33.1 | - | - | 25.2 | EN | [Pramanik et al.](https://arxiv.org/abs/2108.08614) | ❌ | +| QAnswer | 2020 | 30.80 | - | - | - | EN | [Pramanik et al.](https://arxiv.org/abs/2108.08614) | ❌ | +| GraftNet | 2018 | 19.7 | - | - | - | EN | [Christmann P. et al](https://arxiv.org/pdf/2108.08597.pdf) | ❌ | +| ElNeuQA-ConvS2S [1] | 2021 | 26.90 | 27 | 26.90 | - | EN | [Diomedi, Hogan](https://arxiv.org/pdf/2107.02865.pdf) | ❌ | +| GRAFT-Net + Clocq [2] | 2022 | 19.70 | - | - | - | EN | [Christmann P. et al](https://arxiv.org/pdf/2108.08597.pdf) | ❌ | +| Platypus | 2018 | 3.6 | - | - | - | EN | [Pramanik et al.](https://arxiv.org/abs/2108.08614) | ❌ | +| Pullnet | 2019 | 1.1 | - | - | 11.9 | EN | [Pramanik et al.](https://arxiv.org/abs/2108.08614) | ❌ | +| UNIK-QA | 2020 | 0.5 | - | - | - | EN | [Pramanik et al.](https://arxiv.org/abs/2108.08614) | ❌ | +| GETT-QA [4] | 2023 | 40.3 | - | - | - | EN | [Banerjee et al.](https://arxiv.org/pdf/2303.13284.pdf) | ❌ | diff --git a/freebase/WebQuestionsSP.md b/freebase/WebQuestionsSP.md index adfc4211..60583bdf 100644 --- a/freebase/WebQuestionsSP.md +++ b/freebase/WebQuestionsSP.md @@ -6,7 +6,7 @@ | Model / System | Year | F1 | Hits@1 | Accuracy | Language | Reported by | | :---------------------------------: | :--: | :--------: | :--------: | :------: | :------: | :---------------------------------------------------------------------------------------: | | chatGPT | 2023 | - | - | 83.70 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | -| FRED | 2023 | 0.86 ± 0.05| - | - | EN |[Lamott et al.](https://recap.uni-trier.de/static/377a488cc4cee95714b3ad713aa22fa7/88.pdf) | +| FRED | 2023 | 86 ± 5 | - | - | EN |[Lamott et al.](https://recap.uni-trier.de/static/377a488cc4cee95714b3ad713aa22fa7/88.pdf) | | TIARA | 2022 | 78.9 | 75.2 | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) | | DECAF (DPR + FiD-3B) | 2022 | 78.8 | 82.1 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) | | GPT-3.5v3 | 2023 | - | - | 79.60 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | diff --git a/frontend/src/app.postcss b/frontend/src/app.postcss index 48ef964a..12ddf218 100644 --- a/frontend/src/app.postcss +++ b/frontend/src/app.postcss @@ -16,3 +16,15 @@ a { .swal2-container { z-index: 999999 !important; } + +#copy-citation-button { +box-shadow: 0px 4px 6px rgba(0, 0, 0, 0.1); +transition: box-shadow 0.3s ease; +background-color: hsl(rgb(0,0,0) / var(--tw-bg-opacity)); +border-color: hsl(rgb(0,0,0) / var(--tw-bg-opacity)); +width: 400px; +height: 4rem; +margin-top: 2rem; +margin-bottom: 2rem; +} + diff --git a/frontend/src/lib/constants.ts b/frontend/src/lib/constants.ts index a3ede9ba..f049b62f 100644 --- a/frontend/src/lib/constants.ts +++ b/frontend/src/lib/constants.ts @@ -1,19 +1,11 @@ export const PUBLIC_REPO_URL = 'KGQA/leaderboard'; export const NUMERIC_FIELDS = [ - 'Accuracy', - 'Exact Match', - 'F-Score', - 'Precision@1', - 'Precision@5', - 'Hits@1', - 'Hits@5', - 'MRR', - 'EM', - 'Lang_En', - 'Lang_He', - 'Lang_Kn', - 'Lang_Zh' + 'Accuracy', 'Exact Match', 'F-Score', 'Precision@1', 'Precision@5', 'Hits@1', + 'Hits@5', 'MRR', 'EM', 'Lang_En', 'Lang_He', 'Lang_Kn', 'Lang_Zh' ]; -export const GITHUB_BOT_URL = 'https://kgqa-leaderboard-bot.nliwod.org/make_pull_request'; +export const GITHUB_BOT_URL = + 'https://kgqa-leaderboard-bot.nliwod.org/make_pull_request'; +// export const GITHUB_BOT_URL = +// 'https://api.leaderboard.agsolutions.dev/make_pull_request'; diff --git a/frontend/src/routes/+page.svelte b/frontend/src/routes/+page.svelte index 5fc25051..91f55a8b 100644 --- a/frontend/src/routes/+page.svelte +++ b/frontend/src/routes/+page.svelte @@ -99,6 +99,7 @@
KGQA Leaderboard
-
+
-
Or select Knowledge Graph from the list below
+
Or select a Knowledge Graph from the list below
{#if prefaceData}
{#each prefaceData.knowledgeGraphs as kg} @@ -157,14 +158,7 @@
{/if}
- + Systems
{#if content} diff --git a/other/MetaQA - 1 Hop.md b/other/MetaQA - 1 Hop.md index 3d74a981..aedb27e8 100644 --- a/other/MetaQA - 1 Hop.md +++ b/other/MetaQA - 1 Hop.md @@ -18,6 +18,7 @@ | GlobalGraph | 2022 | 99.0 | 97.6 | - | EN | [Zhang et al.](https://downloads.hindawi.com/journals/cin/2022/4734179.pdf) | | 2HR-DR | 2022 | 98.8 | 97.3 | - | EN | [Zhang et al.](https://downloads.hindawi.com/journals/cin/2022/4734179.pdf) | | SGReader | 2022 | 96.7 | 96.0 | - | EN | [Zhang et al.](https://downloads.hindawi.com/journals/cin/2022/4734179.pdf) | +| TERP | 2022 | 97.5 | - | - | EN | [Qiao et al.](https://aclanthology.org/2022.coling-1.156.pdf) | | GRAFT-Net | 2022 | 97.4 | 91.0 | - | EN | [Zhang et al.](https://downloads.hindawi.com/journals/cin/2022/4734179.pdf) | | ARN_DistMult | 2023 | 97.12 | - | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | | ARN_TuckER | 2023 | 97.11 | - | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | diff --git a/other/MetaQA - 2 Hop.md b/other/MetaQA - 2 Hop.md index 1bbdef36..ebc5fa1d 100644 --- a/other/MetaQA - 2 Hop.md +++ b/other/MetaQA - 2 Hop.md @@ -15,6 +15,7 @@ | QNRKGQA | 2022 | 99.9 | - | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) | | QNRKGQA+h | 2022 | 99.9 | - | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) | | SSKGQA | 2022 | 99.7 | - | - | EN | [Mingchen Li and Jonathan Shihao Ji](https://arxiv.org/pdf/2204.10194.pdf) | +| TERP | 2022 | 99.4 | - | - | EN | [Qiao et al.](https://aclanthology.org/2022.coling-1.156.pdf) | | EmbedKGQA | 2020 | 98.8 | - | - | EN | [Saxena et al.](https://aclanthology.org/2020.acl-main.412.pdf) | | TransferNet | 2022 | 98.5 | - | - | EN | [Mingchen Li and Jonathan Shihao Ji](https://arxiv.org/pdf/2204.10194.pdf) | | NRQA | 2022 | 97.5 | - | - | EN | [Guo et al.](https://link.springer.com/content/pdf/10.1007/s10489-022-03927-0.pdf) | diff --git a/other/MetaQA - 3 Hop.md b/other/MetaQA - 3 Hop.md index ecef0537..cbe8a777 100644 --- a/other/MetaQA - 3 Hop.md +++ b/other/MetaQA - 3 Hop.md @@ -12,6 +12,7 @@ | NSM+h | 2021 | 98.9 | - | - | EN | [He et al.](https://arxiv.org/pdf/2101.03737.pdf) | | QNRKGQA | 2022 | 98.9 | - | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) | | QNRKGQA+h | 2022 | 98.9 | - | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) | +| TERP | 2022 | 98.9 | - | - | EN | [Qiao et al.](https://aclanthology.org/2022.coling-1.156.pdf) | | KGQA Based on Query Path Generation | 2022 | 98.5 | - | - | EN | [Yang et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_12)| | QAGCN | 2022 | 97.6 | - | - | EN | [Wang et al.](https://arxiv.org/pdf/2206.01818.pdf) | | ARN_ConvE | 2023 | 97.06 | - | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) | diff --git a/systems.md b/systems.md index 0176ace0..943a9811 100644 --- a/systems.md +++ b/systems.md @@ -139,4 +139,5 @@ | TIARA | Shu et al. | [Link](https://aclanthology.org/2022.emnlp-main.555.pdf) | yes | [Link](https://github.com/microsoft/KC/tree/main/papers/TIARA) | [Link](https://aclanthology.org/2022.emnlp-main.555.pdf) | In this paper, we present a new KBQA model, TIARA, which addresses those issues by applying multi-grained retrieval to help the PLM focus on the most relevant KB contexts, viz., entities, exemplary logical forms, and schema items. Moreover, constrained decoding is used to control the output space and reduce generation errors. | Shu et al. | | MACRE | Xu et al. | [Link](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_40) | no | - | [Link](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_40) | MACRE is a novel approach for multi-hop question answering over KGs via contrastive relation embedding (MACRE) powered by contrastive relation embedding and context-aware relation ranking. | Xu et al. | | KGQAcl/rr | Hu et al. | [Link](https://arxiv.org/pdf/2303.10368.pdf) | yes | [Link](https://github.com/HuuuNan/PLMs-in-Practical-KBQA) | [Link](https://arxiv.org/pdf/2303.10368.pdf) | KGQA-CL and KGQA-RR are tow frameworks proposed to evaluate the PLM's performance in comparison to their efficiency. Both architectures are composed of mention detection, entity disambiguiation, relation detection and answer query building. The difference lies on the relation detection module. KGQA-CL aims to map question intent to KG relations. While KGQA-RR ranks the related relations to retrieve the answer entity. Both frameworks are tested on common PLM, distilled PLMs and knowledge-enhanced PLMs and achieve high performance on three benchmarks. | Hu et al. | -| FRED | Lamott et al. | [Link](https://recap.uni-trier.de/static/377a488cc4cee95714b3ad713aa22fa7/88.pdf) | no | - | [Link](https://recap.uni-trier.de/static/377a488cc4cee95714b3ad713aa22fa7/88.pdf) | Fred combines pattern-based entity retrieval with a transformer-based question encoder. The method uses an evolutionary approach to learn SPARQL patterns, which retrieve candidate entities from a knowledge base. The transformer-based regressor is then trained to estimate each pattern’s expected F1 score for answering the question, resulting in a ranking of candidate entities. Unlike other approaches, FRED can attribute results to learned SPARQL patterns, making them more interpretable. | Lamott et al. | +| FRED | Lamott et al. | [Link](https://recap.uni-trier.de/static/377a488cc4cee95714b3ad713aa22fa7/88.pdf) | no | - | [Link](https://recap.uni-trier.de/static/377a488cc4cee95714b3ad713aa22fa7/88.pdf) | FRED combines pattern-based entity retrieval with a transformer-based question encoder. The method uses an evolutionary approach to learn SPARQL patterns, which retrieve candidate entities from a knowledge base. The transformer-based regressor is then trained to estimate each pattern’s expected F1 score for answering the question, resulting in a ranking of candidate entities. Unlike other approaches, FRED can attribute results to learned SPARQL patterns, making them more interpretable. | Lamott et al. | +| W. Han et al. | Han et al. | [Link](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_39) | no | - | [Link](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_39) | This model is based on machine reading comprehension. To transform a subgraph of the KG centered on the topic entity into text, the subgraph is sketched through a carefully designed schema tree, which facilitates the retrieval of multiple semantically-equivalent answer entities. Then, the promising paragraphs containing answers are picked by a contrastive learning module. Finally, the answer entities are delivered based on the answer span that is detected by the machine reading comprehension module. | Han et al. |