SemEval-2021 Task 11 Shared Task Dataset

NLPContributionGraph - Structuring Scholarly NLP Contributions in the Open Research Knowledge Graph

Background

NLPContributionGraph was introduced as Task 11 at SemEval 2021 for the first time. The task is defined on a dataset of Natural Language Processing (NLP) scholarly articles with their contributions structured to be integrable within Knowledge Graph infrastructures such as the Open Research Knowledge Graph. The structured contribution annotations are provided as (1) Contribution sentences : a set of sentences about the contribution in the article; (2) Scientific terms and relations: a set of scientific terms and relational cue phrases extracted from the contribution sentences; and (3) Triples: semantic statements that pair scientific terms with a relation, modeled toward subject-predicate-object RDF statements for KG building. The Triples are organized under three (mandatory) or more of twelve total information units (viz., ResearchProblem, Approach, Model, Code, Dataset, ExperimentalSetup, Hyperparameters, Baselines, Results, Tasks, Experiments, and AblationAnalysis).

The Shared Task

As a complete submission for the Shared Task, given NLP scholarly articles in plaintext format, systems had to automatically extract the following information: contribution sentences; scientific term and predicate phrases from the sentences; and * (subject,predicate,object) triple statements toward KG building organized under three or more of twelve total information units.