This is one of the "OntologySummit2013_Hackathon_Clinics" Projects ...

(HC-02) The General Ontology Evaluation Framework (GOEF) & the I-Choose Use Case

Project Champions: Joanne Luciano, James Michaelis, NicolauDePaula, DjokoSigitSayogo,

Event Date / Time: (Day-2) Sat 2013.04.06 - Open webcast segment - 2-Hrs. - starting: 8:00am PDT / 11:00am EDT / 5:00pm CEST / 15:00 UTC

Proceedings: [ Presentation ] . [ chat-transcript ] . [ audio archive ( 1:52:36 ; mp3 ; 12.89 MB ) ]

Participant Volunteers:

Project Overview

1. Challenges of Ontology Evaluation

Use case driven ontology evaluation is managed through direct inspection by subject matter experts. However, this is a time-consuming effort, which requires individual review of potentially multiple ontologies. Thus, what if we could develop a system which could take in a use case formalism, and give recommendations for ontologies to use?

2. The General Ontology Evaluation Framework (GOEF)

We would like to use the opportunity of this Hackathon to assist in developing the General Ontology Evaluation Framework (GOEF). We will use I-Choose use case and ontology as an example.

For GOEF:

Goal: Enable objective evaluation of an ontology with respect to a use case.
Both are constructed / deconstructed to extract / expose the evaluation criteria and the ontology-encoded knowledge.
Facilitates ontology design, modular construction, development management, and evaluation is built into the development process.
Conclusion: We need to create procedures, processes, methods to help define, adjudicate, and ensure quality of knowledge capture/representation

3. Stages of the GOEF Approach

The GOEF approach consists of two stages:

1) Recast use case into its components:

Functional objective
Design objective and requirements specification
Semantic components required to achieve above

2) Evaluate components using objective metrics

Place existing evaluation methods in context by utility

4. Motivating Example - I-Choose

a. Short description of I-Choose:

The objective of I-Choose project is to develop an interoperable data framework to provide consumers with a wide range of information about how, where, and by whom products are manufactured and brought to market. More specifically, I-Choose ontology based architecture should enable consumer advocates to retrieve and verify information acquired from social and environmental certification procedures, such as inspections undertaken to acquire FairTrade and Organic certifications. The I-Choose network and ontology team have decided to focus first on an ontology based on FairTrade coffee certification. This preliminary ontology would allow for a prototype application to be developed and tested with an actual (or artificial) dataset. Ultimately, the i-Choose project is interested in developing a data standard for this sort of inspection/certification.

b. Envision Mock-up Application of I-Choose

The image above shows a mock up application leveraged on a proposed I-Choose architecture. In this application, a user would be able to retrieve "extra" information about a product not provided in its package. The I-Choose project had thus set out to map the entire domain of coffee production, distribution, and consumption. This would have enabled the consumer or consumer advocate to retrieve a number of data points on characteristics of the particular product he/she would be interested in purchasing. To enable this envisioned application, I-Choose architecture would need to gather data from 3rd party certifiers and supply chain companies.

Preliminary research by I-Choose ontology team has shifted the focus of the I-Choose project to target data obtained through inspection/certification processes for sustainable certified coffee as a starting point.

Overview of Sustainability Certification Schemes

A certification system generally consists of a standard setter and a certification body consists of a standard setter and a certification body. One of the fair trade standard setters is Fairtrade International (FLO). FLO creates standards and manages the labeling initiative.

The standard setter relies on an independent certification body to verify compliance to the standard. The certification body for FLO is FLO-CERT, an internal body but independent to FLO. Flo-cert interprets the standard into verifiable control points called compliance criteria. Other sustainable certification schemes may use independent an external certification body (such as: UTZ Good Inside) or use a non-independent internal certification body (such as: Rainforest Action Network/RAN).

One of the fair trade standard setters is Fairtrade International (FLO). FLO creates standards and manages the labeling initiative. The standard setter relies on an independent certification body to verify compliance to the standard. The certification body for FLO is FLO-CERT, an internal body but independent to FLO. FLO-CERT interprets the standard into verifiable control points called compliance criteria. Other sustainable certification schemes may use independent an external certification body (such as: UTZ Good Inside) or use a non-independent internal certification body (such as: Rainforest Action Network/RAN).

An applicant wishing to receive the Fairtrade (FLO) certification will be evaluated against the compliance criteria by an inspector/auditor appointed by Flo-cert. Inspector/Auditor will follow each compliance criteria specified by the certification body-applicable to the entity being evaluated (e.g. small producer organization). The result from the audit/inspection will be used for the ultimate certification decision which, in this case, will grant the particular Fairtrade International certification mark/label.

c. Preliminary lower level ontology of criteria evaluation

This partial, preliminary lower level ontology pictured above defines the class of "CriteriaEvaluation". Child classes of the CriteriaEvaluation class are the specific evaluations of each compliance criterion. In this ontology each criterion evaluation is modeled as a class. We believe the domain of certification/inspection is similar to pollution evaluation, and thus began to model our ontology after the Tetherless World Semanteco Ontology(http://tw.rpi.edu/web/project/SemantEco). As noted above (4.d.2) each compliance criterion evaluation has a number of characteristics (data elements) associated with it.

d. The I-Choose Use Case

Use Case Name:

Customer Advocate verifying compliance to Fairtrade certification criteria

Scenario - Retrieving Workers Welfare and Child Labor Evaluation

Goal:

A customer might want to know about firm's compliance to specific Fairtrade certification criteria. In this scenario the criteria of interest are workers welfare and/or child labor. As result, a consumer advocate needs to be able to verify the certification criteria evaluations/compliance by retrieving specific criteria evaluations pertinent to the topic of interest.

Overview:

Consumer advocate (CA) needs to verify the compliance of producer organization (PO) "XX" to Fairtrade criteria pertinent to workers welfare and no child labor practices. The reason being, there are several queries from consumers who are asking for verification of the workers welfare and child labor practice by PO "XX". In relation, CA needs to retrieve the Fairtrade certification results of PO "XX" including the score on criteria for workers' welfare and no child labor practices. To do so, an understanding of how Fairtrade compliance criteria work is critical. Including the role of each instance in the compliance criteria contribute to the certification decision as well as the actors involved in the certification process.

Actors involved in certification decision:

In general, a certification system consists of a standard setter and a certification body. One "fair trade" standard setter is Fairtrade International (FLO). The certification body for FLO is FLO-CERT, an internal body but independent to FLO. FLO creates standards and manage the labeling initiative. Flo-cert interpret standard into verifiable control points called compliance criteria. An applicant to Fairtrade certification will be evaluated against the compliance criteria by an inspector/auditor appointed by Flocert.

List of actors:

Standard Setter Organization - Fair Trade International.

Certification Body - FLO-Cert

Inspector / Auditor - named_auditor

Small Farmer Organizations - named_coop

5. I-Choose under GOEF

Function:

Enable retrieval of specific criteria evaluations that occurred during an evaluation process of a particular product.

Design objective:

Initial system: Satisfy consensus user criteria pre-determined by survey research

Semantic components:

Compliance Criteria

a) Pesticide
b) Minimum Wage
c) Child labor

Standard

a) FairTrade International
b) USDA Organic
c) Sustainable Agriculture Alliance

Certification Body

a) Flo-Cert
b) Certified private inspectors
c) Sustainable Farm Certification Intl, Ltd.

Product

a) Coffee
b) Sugar Cane
c) Fruit

Evaluation Metrics:

Correctness

General logical/syntactical validation
Are the right terms used (compliance criteria vs. guidelines vs. standards)
Match information provided in the ontology to information consensus user wants (surveyed).

Completeness

Calculate % coverage of minimum terms
All "severe" pesticides listed (certain %)
All pesticides prohibited by U.S. EPA. Listed

Utility

Validate against known test sets
Consumer Consensus Questions Satisfied

6. Final remarks

We have presented a method to evaluate ontologies, namely the Generalized Ontology Evaluation Framework (GOEF). We hope that the Hackathon Ontology Clinic will assist is systematizing the necessary elements of this approach.

Ontolog Forum

Contents