Skip to content

Commit

Permalink
Merge branch 'v2.0' into issue_172
Browse files Browse the repository at this point in the history
  • Loading branch information
@DavidBlavid
DavidBlavid authored Oct 31, 2023
2 parents c156a6e + 18863cd commit a628bc9
Show file tree
Hide file tree
Showing 9 changed files with 61 additions and 58 deletions.
61 changes: 31 additions & 30 deletions dbpedia/LC-QuAD v2.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,33 +3,34 @@ name: LC-QuAD v2
datasetUrl: https://figshare.com/projects/LCQuAD_2_0/62270
---

| Model / System | Year | Precision | Recall | F1 | Language | Reported by | Gold Entity |
| :-----------------------: | :--: | :-------: | :----: | :---: | :------: | :----------------------------------------------------------------------------------: | :---------: |
| T5-Small | 2022 | - | - | 92 | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ |
| T5-Base | 2022 | - | - | 91 | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ |
| PGN-BERT-BERT | 2022 | - | - | 86 | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ |
| SGPT_Q,K [1] | 2022 | - | - | 89.04 | EN | [Al Hasan Rony et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9815253) | ✅ |
| PGN-BERT | 2022 | - | - | 77 | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ |
| NSpM [2] | 2022 | - | - | 66.47 | EN | [Al Hasan Rony et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9815253) | ✅ |
| BART | 2022 | - | - | 64 | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ |
| Zou et al. + Bert | 2021 | - | - | 59.30 | EN | [Zou et al.](https://arxiv.org/pdf/2111.06086.pdf) | ✅ |
| CLC | 2021 | - | - | 59 | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ |
| Multi-hop QGG | 2020 | - | - | 53 | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ |
| Zou et al. + Tencent Word | 2021 | - | - | 52.90 | EN | [Zou et al.](https://arxiv.org/pdf/2111.06086.pdf) | ✅ |
| Multi-hop QGG | 2021 | - | - | 52.60 | EN | [Zou et al.](https://arxiv.org/pdf/2111.06086.pdf) | ✅ |
| AQG-net | 2021 | - | - | 44.90 | EN | [Zou et al.](https://arxiv.org/pdf/2111.06086.pdf) | ✅ |
| SGPT_Q [3] | 2022 | - | - | 83.45 | EN | [Al Hasan Rony et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9815253) | ❌ |
| ChatGPT | 2023 | - | - | 42.76 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | ❌ |
| GPT-3.5v3 | 2023 | - | - | 39.04 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | ❌ |
| GPT-3.5v2 | 2023 | - | - | 33.77 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | ❌ |
| GPT-3 | 2023 | - | - | 33.04 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | ❌ |
| FLAN-T5 | 2023 | - | - | 30.14 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | ❌ |
| UNIQORN | 2021 | 33.1 | - | - | EN | [Pramanik et al.](https://arxiv.org/abs/2108.08614) | ❌ |
| QAnswer | 2020 | 30.80 | - | - | EN | [Pramanik et al.](https://arxiv.org/abs/2108.08614) | ❌ |
| GraftNet | 2018 | 19.7 | - | - | EN | [Christmann P. et al](https://arxiv.org/pdf/2108.08597.pdf) | ❌ |
| ElNeuQA-ConvS2S [1] | 2021 | 26.90 | 27 | 26.90 | EN | [Diomedi, Hogan](https://arxiv.org/pdf/2107.02865.pdf) | ❌ |
| GRAFT-Net + Clocq [2] | 2022 | 19.70 | - | - | EN | [Christmann P. et al](https://arxiv.org/pdf/2108.08597.pdf) | ❌ |
| Platypus | 2018 | 3.6 | - | - | EN | [Pramanik et al.](https://arxiv.org/abs/2108.08614) | ❌ |
| Pullnet | 2019 | 1.1 | - | - | EN | [Pramanik et al.](https://arxiv.org/abs/2108.08614) | ❌ |
| UNIK-QA | 2020 | 0.5 | - | - | EN | [Pramanik et al.](https://arxiv.org/abs/2108.08614) | ❌ |
| GETT-QA [4] | 2023 | 40.3 | - | - | EN | [Banerjee et al.](https://arxiv.org/pdf/2303.13284.pdf) | ❌ |
| Model / System | Year | Precision | Recall | F1 | Hits@1 | Language | Reported by | Gold Entity |
| :-----------------------: | :--: | :-------: | :----: | :---: | :----: | :------: | :----------------------------------------------------------------------------------: | :---------: |
| T5-Small | 2022 | - | - | 92 | - | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ |
| T5-Base | 2022 | - | - | 91 | - | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ |
| PGN-BERT-BERT | 2022 | - | - | 86 | - | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ |
| SGPT_Q,K [1] | 2022 | - | - | 89.04 | - | EN | [Al Hasan Rony et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9815253) | ✅ |
| PGN-BERT | 2022 | - | - | 77 | - | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ |
| NSpM [2] | 2022 | - | - | 66.47 | - | EN | [Al Hasan Rony et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9815253) | ✅ |
| BART | 2022 | - | - | 64 | - | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ |
| Zou et al. + Bert | 2021 | - | - | 59.30 | - | EN | [Zou et al.](https://arxiv.org/pdf/2111.06086.pdf) | ✅ |
| CLC | 2021 | - | - | 59 | - | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ |
| Multi-hop QGG | 2020 | - | - | 53 | - | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ |
| Zou et al. + Tencent Word | 2021 | - | - | 52.90 | - | EN | [Zou et al.](https://arxiv.org/pdf/2111.06086.pdf) | ✅ |
| Multi-hop QGG | 2021 | - | - | 52.60 | - | EN | [Zou et al.](https://arxiv.org/pdf/2111.06086.pdf) | ✅ |
| AQG-net | 2021 | - | - | 44.90 | - | EN | [Zou et al.](https://arxiv.org/pdf/2111.06086.pdf) | ✅ |
| SGPT_Q [3] | 2022 | - | - | 83.45 | - | EN | [Al Hasan Rony et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9815253) | ❌ |
| W. Han et al. | 2023 | - | - | - | 42.6 | EN | [Han et al.](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_39) | ❌ |
| ChatGPT | 2023 | - | - | 42.76 | - | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | ❌ |
| GPT-3.5v3 | 2023 | - | - | 39.04 | - | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | ❌ |
| GPT-3.5v2 | 2023 | - | - | 33.77 | - | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | ❌ |
| GPT-3 | 2023 | - | - | 33.04 | - | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | ❌ |
| FLAN-T5 | 2023 | - | - | 30.14 | - | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | ❌ |
| UNIQORN | 2021 | 33.1 | - | - | 25.2 | EN | [Pramanik et al.](https://arxiv.org/abs/2108.08614) | ❌ |
| QAnswer | 2020 | 30.80 | - | - | - | EN | [Pramanik et al.](https://arxiv.org/abs/2108.08614) | ❌ |
| GraftNet | 2018 | 19.7 | - | - | - | EN | [Christmann P. et al](https://arxiv.org/pdf/2108.08597.pdf) | ❌ |
| ElNeuQA-ConvS2S [1] | 2021 | 26.90 | 27 | 26.90 | - | EN | [Diomedi, Hogan](https://arxiv.org/pdf/2107.02865.pdf) | ❌ |
| GRAFT-Net + Clocq [2] | 2022 | 19.70 | - | - | - | EN | [Christmann P. et al](https://arxiv.org/pdf/2108.08597.pdf) | ❌ |
| Platypus | 2018 | 3.6 | - | - | - | EN | [Pramanik et al.](https://arxiv.org/abs/2108.08614) | ❌ |
| Pullnet | 2019 | 1.1 | - | - | 11.9 | EN | [Pramanik et al.](https://arxiv.org/abs/2108.08614) | ❌ |
| UNIK-QA | 2020 | 0.5 | - | - | - | EN | [Pramanik et al.](https://arxiv.org/abs/2108.08614) | ❌ |
| GETT-QA [4] | 2023 | 40.3 | - | - | - | EN | [Banerjee et al.](https://arxiv.org/pdf/2303.13284.pdf) | ❌ |
2 changes: 1 addition & 1 deletion freebase/WebQuestionsSP.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
| Model / System | Year | F1 | Hits@1 | Accuracy | Language | Reported by |
| :---------------------------------: | :--: | :--------: | :--------: | :------: | :------: | :---------------------------------------------------------------------------------------: |
| chatGPT | 2023 | - | - | 83.70 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) |
| FRED | 2023 | 0.86 ± 0.05| - | - | EN |[Lamott et al.](https://recap.uni-trier.de/static/377a488cc4cee95714b3ad713aa22fa7/88.pdf) |
| FRED | 2023 | 86 ± 5 | - | - | EN |[Lamott et al.](https://recap.uni-trier.de/static/377a488cc4cee95714b3ad713aa22fa7/88.pdf) |
| TIARA | 2022 | 78.9 | 75.2 | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) |
| DECAF (DPR + FiD-3B) | 2022 | 78.8 | 82.1 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) |
| GPT-3.5v3 | 2023 | - | - | 79.60 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) |
Expand Down
12 changes: 12 additions & 0 deletions frontend/src/app.postcss
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,15 @@ a {
.swal2-container {
z-index: 999999 !important;
}

#copy-citation-button {
box-shadow: 0px 4px 6px rgba(0, 0, 0, 0.1);
transition: box-shadow 0.3s ease;
background-color: hsl(rgb(0,0,0) / var(--tw-bg-opacity));
border-color: hsl(rgb(0,0,0) / var(--tw-bg-opacity));
width: 400px;
height: 4rem;
margin-top: 2rem;
margin-bottom: 2rem;
}

20 changes: 6 additions & 14 deletions frontend/src/lib/constants.ts
View file
Original file line number Diff line number Diff line change
@@ -1,19 +1,11 @@
export const PUBLIC_REPO_URL = 'KGQA/leaderboard';

export const NUMERIC_FIELDS = [
'Accuracy',
'Exact Match',
'F-Score',
'Precision@1',
'Precision@5',
'Hits@1',
'Hits@5',
'MRR',
'EM',
'Lang_En',
'Lang_He',
'Lang_Kn',
'Lang_Zh'
'Accuracy', 'Exact Match', 'F-Score', 'Precision@1', 'Precision@5', 'Hits@1',
'Hits@5', 'MRR', 'EM', 'Lang_En', 'Lang_He', 'Lang_Kn', 'Lang_Zh'
];

export const GITHUB_BOT_URL = 'https://kgqa-leaderboard-bot.nliwod.org/make_pull_request';
export const GITHUB_BOT_URL =
'https://kgqa-leaderboard-bot.nliwod.org/make_pull_request';
// export const GITHUB_BOT_URL =
// 'https://api.leaderboard.agsolutions.dev/make_pull_request';
18 changes: 6 additions & 12 deletions frontend/src/routes/+page.svelte
View file
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,7 @@
<div class="text-center mt-4 text-5xl">KGQA Leaderboard</div>
<div class="flex justify-center">
<button
id="copy-citation-button"
on:click={() => {
let citation =
'[Knowledge Graph Question Answering Leaderboard: A Community Resource to Prevent a Replication Crisis](https://aclanthology.org/2022.lrec-1.321) (Perevalov et al., LREC 2022)';
Expand All @@ -112,11 +113,11 @@
</button>
</div>

<div class="flex justify-center">
<div class="flex justify-center mt-[3rem]">
<input
type="text"
class="input input-primary input-sm md:w-[25%] w-11/12"
placeholder="Search Datasets 🔎"
class="input input-primary input-sm md:w-[25%] w-[400px]"
placeholder="🔎 Search Datasets"
name="filter"
id="filter"
on:input={search}
Expand Down Expand Up @@ -146,7 +147,7 @@
{/if}
</div>
</div>
<div class="w-[70%] mx-auto divider">Or select Knowledge Graph from the list below</div>
<div class="w-[70%] mx-auto divider mt-[2rem]">Or select a Knowledge Graph from the list below</div>
{#if prefaceData}
<div transition:fade|local class="flex justify-center space-x-4">
{#each prefaceData.knowledgeGraphs as kg}
Expand All @@ -157,14 +158,7 @@
</div>
{/if}
<div class="flex justify-center my-2">
<button
on:click={() => {
window.location.href = base + '/systems';
}}
class="btn btn-accent"
>
Systems
</button>
<a class="" href={base + '/systems'}>Systems</a>
</div>
<div class="my-10" />
{#if content}
Expand Down
1 change: 1 addition & 0 deletions other/MetaQA - 1 Hop.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
| GlobalGraph | 2022 | 99.0 | 97.6 | - | EN | [Zhang et al.](https://downloads.hindawi.com/journals/cin/2022/4734179.pdf) |
| 2HR-DR | 2022 | 98.8 | 97.3 | - | EN | [Zhang et al.](https://downloads.hindawi.com/journals/cin/2022/4734179.pdf) |
| SGReader | 2022 | 96.7 | 96.0 | - | EN | [Zhang et al.](https://downloads.hindawi.com/journals/cin/2022/4734179.pdf) |
| TERP | 2022 | 97.5 | - | - | EN | [Qiao et al.](https://aclanthology.org/2022.coling-1.156.pdf) |
| GRAFT-Net | 2022 | 97.4 | 91.0 | - | EN | [Zhang et al.](https://downloads.hindawi.com/journals/cin/2022/4734179.pdf) |
| ARN_DistMult | 2023 | 97.12 | - | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) |
| ARN_TuckER | 2023 | 97.11 | - | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) |
Expand Down
1 change: 1 addition & 0 deletions other/MetaQA - 2 Hop.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
| QNRKGQA | 2022 | 99.9 | - | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) |
| QNRKGQA+h | 2022 | 99.9 | - | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) |
| SSKGQA | 2022 | 99.7 | - | - | EN | [Mingchen Li and Jonathan Shihao Ji](https://arxiv.org/pdf/2204.10194.pdf) |
| TERP | 2022 | 99.4 | - | - | EN | [Qiao et al.](https://aclanthology.org/2022.coling-1.156.pdf) |
| EmbedKGQA | 2020 | 98.8 | - | - | EN | [Saxena et al.](https://aclanthology.org/2020.acl-main.412.pdf) |
| TransferNet | 2022 | 98.5 | - | - | EN | [Mingchen Li and Jonathan Shihao Ji](https://arxiv.org/pdf/2204.10194.pdf) |
| NRQA | 2022 | 97.5 | - | - | EN | [Guo et al.](https://link.springer.com/content/pdf/10.1007/s10489-022-03927-0.pdf) |
Expand Down
1 change: 1 addition & 0 deletions other/MetaQA - 3 Hop.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
| NSM+h | 2021 | 98.9 | - | - | EN | [He et al.](https://arxiv.org/pdf/2101.03737.pdf) |
| QNRKGQA | 2022 | 98.9 | - | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) |
| QNRKGQA+h | 2022 | 98.9 | - | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) |
| TERP | 2022 | 98.9 | - | - | EN | [Qiao et al.](https://aclanthology.org/2022.coling-1.156.pdf) |
| KGQA Based on Query Path Generation | 2022 | 98.5 | - | - | EN | [Yang et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_12)|
| QAGCN | 2022 | 97.6 | - | - | EN | [Wang et al.](https://arxiv.org/pdf/2206.01818.pdf) |
| ARN_ConvE | 2023 | 97.06 | - | - | EN | [Cui et al.](https://www.sciencedirect.com/science/article/abs/pii/S0020025522013317) |
Expand Down
Loading

0 comments on commit a628bc9

Please sign in to comment.