Results obtained by the CSR tool can be found here!


Ontology Alignment Evaluation Initiative - OAEI-2011 CampaignOAEI

Oriented Matching 2011

An increasing number of matchers are now capable of deriving mapping relations other than equivalence relations, such as subsumption, disjointness or named relations. This is a necessity given that we need to compute alignments between ontologies at different granularity levels or between ontologies that elaborate on non-equivalent elements. The evaluation of such mappings was addressed already in OAEI (2009) Oriented Matching track. Following these goals we wish to get a better insight into the types of non-equivalence mappings that state-of-the-art tools produce and report on existing methods for computing subsumption mappings. The track aims also to report on evaluation methods and measures for subsumption mappings, in conjunction to the computation of equivalence mappings. Targeting these goals, we have built new benchmark datasets that are described below. The track focuses on a gold standard based evaluation for two given datasets that has been derived from a) the OAEI benchmark dataset (all possible pairs of the real ontologies 301 to 304) and b) all pairs of ontologies in the Illinois course catalogs dataset.

Data sets

Concerning the evaluation of equivalence and subsumption relations the track provides two datasets for the participants:

Real Ontologies - Academia: This dataset involves ontology pairs given specific alterations of the four real ontologies in the Benchmark series corpus of OAEI, namely, the 301, 302, 303, and 304, dealing with bibliographic references. The alterations aim to introduce subsumption mappings between classes that cannot be inferred via reasoning.

Real Ontologies – Course Catalogs: The dataset involves ontology pairs that are produced from the alterations of four real ontologies concerning description of courses in the universities of Cornell and Washington. Similarly to the Real Ontologies the six pairs include subsumption mappings between classes that cannot be inferred via reasoning.

For each pair of ontologies we provide reference alignments for both equivalence and subsumption mappings and we aim to evaluate tools for their ability to identify both equivalences and subsumptions between classes. However, we aim at distinguishing between tools that (a) compute class equivalences and subsumptions, (b) compute class subsumptions without computing any equivalences and (c) compute class subsumptions and non-class equivalences. These can be done either by declaration of the participants or by providing justifications for the alignments: For subsumption mappings that cannot be inferred (e.g. by exploiting equivalence or subsumption relations between classes/properties), we have added an annotation property called 'Alignment_Argument' in the corresponded alignment cells of the refalign.rdf file. The value of this property is an argument upon the reference alignment. Each tool can assign a value to this property in relation to each mapping relation discovered that provides grounding for its decisions.

It is important to use our related namespace ( in order to annotate your alignments (similar to our refalign.rdf files provided)!!!

The reference alignments (refalign.rdf) have been constructed by manually creating a merged ontology for each pair of source ontologies as follows:

1. Import the two source ontologies 
2. Manually  introduce equivalences between classes of the two imported ontologies
3. Classify classes in the final ontology
4. Introduce additional subsumption relations
5. The resulting ontology is translated to equivalence and subsumption mappings that involve classes with different URIs to the RDF rendered of the Alignment API (RDFRendererVisitor).

The datasets are accompanied with documentation of the specific changes made to the original datasets. The rationale behind these changes can be found a) in the documentation.txt file in each ontology pair folder, b) as an annotation property in the specification of the subsumption relations in the merged ontology, c) in the corresponded alignment cells of the refalign.rdf file.

Datasets can be downloaded directly from here.


We will use precision, recall and f-measures for the efficacy of tools to produce the mappings. Participants will have to provide an alignment containing both equivalent and subsumption relations, optionally (but desired) with corresponding arguments.


May 30th

datasets are out

June 27th

end of commenting period

July 1st

tests are frozen

September 1st

participants send preliminary results for interoperability-checking (old-style tracks), or ensure that the last version of their tools is submitted to the SEALS platform (SEALS tracks, see further instructions for at the Execution phase)

September 23rd

participants send final results (old-style tracks) or submit the ultimate tool version to the SEALS platform (SEALS tracks). All participants must send their papers.

October 22nd

final results ready and OM-2011 workshop.

November 15th

participants send final versions of papers to Cassia Trojahn and Pavel Shvaiko.


The alignArguments namespace extension in order to be able to annotate your alignments (similar to our refalign.rdf files provided) can be found at Additional argumentation extensions, published as part of the on-line Alignment API documentation at, may be used.

- with CSR tool (.pdf)


George Vouros, Konstantinos Kotis and Vassilis Spiliopoulos, University of the Aegean, Greece