OntologySummit2008 - Quality and Gatekeeping

We distinguish between gatekeeping and quality control. Gatekeeping criteria are a set of minimal requirements that any ontology within the OOR has to meet. These criteria are intended to enable the users of the OOR to find ontologies that fit their needs quickly, they are not supposed to ensure the quality of the ontologies.

Gatekeeping Criteria

The ontologies in the OOR have to meet the following criteria.

1. The ontology is open. (see below)

2. The ontology is expressed in a formal language with a well-defined syntax.

3. The authors of the ontology provide the required metadata.

4. The ontology has a clearly specified and clearly delineated scope.

5. Successive versions of an ontology are clearly identified.

6. The ontology is adequately labeled.

So far the most controversial suggested criterion has been the"openness". We need to distinguish between different kinds of "openness", in particular between 'open' development processes and 'open' software licenses. Different members of the community have different preferences on which kinds of openness and how much openness should be required. Some would like to cancel 'openness' as a gateway criterion and rather require the developers of ontologies to provide metadata that allows potential users to understand how 'open' (and in which senses of the word) an ontology is. This issue needs to be addressed during the meeting in Gaithersburg.

Quality Control

The community agrees that it is not sufficient for the OOR just to store ontologies, but that it needs to provide the possibility to evaluate the ontologies within it. There is no agreement on how to evaluate ontologies; the main strategies suggested are: (i) A market driven approach where ontologies are reviewed by users and ranked in a collaborative-filtering or social-network-like process; and (ii) an editorial process where ontologies are reviewed by experts in a similar way as papers which are submitted to scientific journals. The difference in opinion about ontology evaluation reflects the fact that the members of the community are using ontologies for different purposes and thus have different perspectives on what ontologies are. However, there is agreement that the OOR should accept ontologies regardless of whether their developers see ontologies as pieces of software, as representations of scientific knowledge, or as standardized vocabularies. Accordingly, the OOR needs to enable the different styles of evaluation and different standards for ontologies. We suggest a distributed governance model where the OOR allows for subcommunities that provide stewardship for their respective fields by evaluating the available ontologies and by distinguishing high-quality ontologies according to appropriate standards.

-- This page is maintained by: BarrySmith and FabianNeuhaus