Actions

Ontolog Forum

Ontology Summit 2011: Panel Session-4 - (Track-3) "Value Metrics, Value Models and the Value Proposition" - Thu 2011_02_17

Summit Theme: OntologySummit2011: Making the Case for Ontology

Session Title: Value Metrics, Value Models and the Value Proposition - Take I

Session Co-chairs: Dr. ToddSchneider (Raytheon) & Mr. RexBrooks (Starbourne)

Panelists:

  • Mr. RexBrooks (Starbourne) - "Introduction" [ slides ]
  • Mr. KurtConrad (Sagebrush) - "Business Value Alignment to Support Ontology Development" [ slides ]
  • Mr. Rex Brooks / Mr. ChristianFillies (Semtation GmbH) - "Ontology Integration" [ slides ]
  • Ms. MaryBalboni (Raytheon) - "Ontology Performance" [ slides (pdf) ]
  • Dr. JohnYanosy (Rockwell Collins) - "Ontology and Business Value" [ slides ]
  • Dr. ToddSchneider (Raytheon) - "Ontology Use Maintenance" [ slides (pdf) ]

Abstract

OntologySummit2011 Theme: "Making the Case for Ontology"

  • Track-3 Focus: "Value Metrics, Value Models and the Value Proposition"
  • Session Title: Value Metrics, Value Models and the Value Proposition - Take I

This is our 6th Ontology Summit, a joint initiative by NIST, Ontolog, NCOR, NCBO, IAOA & NCO_NITRD. The theme adopted for this Ontology Summit is: "Making the Case for Ontology."

This year's Ontology Summit seeks to address the need to provide concrete evidence of successful deployment of ontologies by examining several application domains for such examples, and in better articulating where different "strengths" of ontological representation are best applied. To support that, the summit also aims to classify the categories of applications where ontology has been, and could be, successfully applied; to identify distinct types of metrics that might be used in evaluating the return on investment in an ontology application (cost, capability, performance, etc.); to lay out some strategies for articulating a case for ontological applications; and to identify remaining challenges and roadblocks to a wider deployment of such applications that represent promising application areas and research challenges for the future. The findings of the summit will be documented in the form of a communiqué intended for public consumption.

The Panel Session today is organized by our Track-3 co-champions, who has assembled an expert panel to help us explore various aspects of Value Metrics, Value Models and the Value Proposition of Applying Ontology.

See developing details on this Summit series of events at: OntologySummit2011 (home page for this summit)

Agenda

Ontology Summit 2011 - Panel Session-4

  • Session Format: this is a virtual session conducted over an augmented conference call

Proceedings

Please refer to the above

IM Chat Transcript captured during the session

see raw transcript here.

(for better clarity, the version below is a re-organized and lightly edited chat-transcript.)

Participants are welcome to make light edits to their own contributions as they see fit.

-- begin in-session chat-transcript --

[09:17] Peter P. Yim: .

Welcome to the Ontology Summit 2011: Panel Session-4

(Track-3) "Value Metrics, Value Models and the Value Proposition" - Thu 2011_02_17

Summit Theme: Ontology Summit 2011: Making the Case for Ontology

Session Title: Value Metrics, Value Models and the Value Proposition - Take I

Session Co-chairs: Dr. Todd Schneider (Raytheon) & Mr. Rex Brooks (Starbourne)

Panelists:

  • Mr. Kurt Conrad (Sagebrush) - "Business Value Alignment to Support Ontology Development"
  • Dr. John Yanosy (Rockwell Collins) - "Ontology and Business Value"

.

[09:22] Peter P. Yim: please refer to session details at: http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2011_02_17

[09:26] Steve Ray: All Hail Watson!

[09:28] Nicola Guarino: Hi everybody

[09:29] Rex Brooks: Hi All

[09:30] anonymous morphed into Bruce Bray

[09:32] Pat Barkman: hi

[09:34] Leo Obrst: Hi, all. My old bearded collie was named Watson, and so I was rooting against the

humans. ;)

[09:34] Bob Smith: Hello, and Rex, your voice is still fuzzy

[09:35] anonymous morphed into Alex Mirzaoff

[09:39] Steve Ray: Cheat sheet: *3 to un-mute; *2 to mute

[09:40] vnc2: session starts

[09:41] vnc2: == Rex Brooks - introduction ==

[09:46] anonymous morphed into Pavithra Kenjige

[09:48] anonymous1 morphed into Kurt Conrad

[09:48] Kurt Conrad: I'm here

[09:48] Steve Ray: Cheat sheet: *3 to un-mute; *2 to mute

[09:49] Steve Ray: == Kurt Conrad presents ==

[09:51] anonymous morphed into Ram Gouripeddi

[09:54] Michael Grüninger: @Kurt: What are the agents in "agent-specific alignments"?

[09:56] Pavithra Kenjige: Actually I could hear you, but a little low voice

[09:56] Michael Grüninger: thanks!

[09:56] anonymous morphed into John Yanosy

[09:56] Steve Ray: We're on slide 4

[09:59] Steve Ray: Maybe it's just me, but if I were a business person trying to decide whether to

invest in an ontological approach, I would be very lost by now.

[10:01] John Yanosy: This slide is very informative with respect to understanding knolwedge sources,

not sure how to incorporate tacit knowledge into ontologies

[10:06] Leo Obrst: @Kurt: there is also a technical notion of "dynamic semantics" for natural

language semantics that goes back to Kamp, Heim in the early 1980s and Groenendijk and Stokhof in

1990, 1991, that has subsequently been developed by others.

[10:04] Steve Ray: == Rex Brooks presents ==

[10:04] Steve Ray: Go ahead. We can use the downloaded version as well.

[10:06] anonymous morphed into Sarah Goldman

[10:08] Pat Barkman: novice here ... anyone want to state what "common tool" he's talking about?

Semaphore/smartlogic?

[10:09] Steve Ray: Not sure why the name wasn't mentioned. It could have been a UML tool like

Enterprise Architect, MagicDraw, or Rational Rose.

[10:09] Pat Barkman: ahhh ... thanks Steve

[10:24] Rex Brooks: @Pat, as far as modeling tools, SemTalk makes Visio into a real modeling tool, at

least for BPMN and several other specific kinds of specialized analytics. It can output XMI and that

allows me to import it into Enterprise Architect that then gives me all the UML I need and then

some. Both SemTalk and EA output images of diagrams and generate code and have dedicated back end

databases. i'm still working on getting some of the conceptualizing into Compendium.

[10:25] Pat Barkman: @Rex ... thanks that helps.

[10:13] Steve Ray: Still anxious to start hearing suggested metrics...cost? development time?

capability? system performance?

[10:15] Pat Barkman: I have the same questions, Steve ... cost/benefit breakdowns

[10:19] Rex Brooks: @Steve, don't worry, its coming.

[10:19] Rex Brooks: @Steve, Since I knew there was all that coming, I felt free to deal with

expectations and focusing on the Value Proposition more than the Value Metrics and Models.

[10:19] Steve Ray: OK

[10:24] Todd Schneider: Steve, are we getting closer?

[10:24] Rex Brooks: hehe

[10:15] Steve Ray: By the way, Todd was very clear just now on the phone.

[10:17] Rex Brooks: @Pat Sorry, SemTalk uses Visio but adds semantic capabilities to it. I happen to

be involved with SemTalk USA and wanted to avoid the appearance of attempting to sell a product.

[10:17] Pat Barkman: thanks Rex

[10:25] Peter P. Yim: @MaryBalboni - ref. your slide#5, is there a maturity model for ontology that you

are using currently?

[10:25] Todd Schneider: Peter, I don't think so.

[10:25] Pat Barkman: ... and yes I'm luvin' the MA here great job Mary!

[10:27] Rex Brooks: @Peter I am pretty sure there isn't one that has much support yet.

[10:29] Peter P. Yim: ref. maturity model, it may be nice to compare what is out there ... e.g. ref.

LeoObrst's OMM which he posted to the [ontology-summit] list yesterday

(http://ontolog.cim3.net/forum/ontology-summit/2011-02/msg00061.html ) - which is at:

http://ontolog.cim3.net/file/work/OntologySummit2011/reference/ontologyMaturityModel-obrst-2009.pdf

[10:29] Steve Ray: @Todd: Yes, I'm breathing a bit easier. So far I'm hearing Performance, and Cost,

with some good expansion of each. The reason I'm asking is because I'm thinking of how this will

integrate into the other tracks - Use cases, and Application framework.

[10:29] Todd Schneider: Steve, understood.

[10:31] Steve Ray: Not sure I would conflate risk and performance...

[10:32] John Yanosy: great job

[10:32] Steve Ray: Great talk by Mary - brings out some good metrics.

[10:33] Mary Balboni: thank you

[10:33] Pat Barkman: yes ... I think that's a jump as well Steve, but given the maturity/prevalance

of ontology implementations ... it's a fair stepping stone in MA

[10:33] Ramdsriram: Are there any case studies (with data) out there which describe how ontologies

improved a system performance?

[10:33] John Yanosy: Todd, I will have to be leaving soon for a customer meeting, approx 15 minutes

[10:33] Todd Schneider: All, we have to save questions. John Yanosy has a hard stop time.

[10:34] Rex Brooks: It's almost time to introduce John.

[10:34] Peter P. Yim: @Todd ... are you doing Q&A after all the panelists have finished their

presentations?

[10:37] Todd Schneider: Peter, yes.

[10:34] Pat Barkman: thanks Mary ... great job ... will be referencing your work in mine, so thanks

[10:35] Pavithra Kenjige: thank you Mary

[10:35] Mary Balboni: thanks everyone - glad to be part of this group

[10:38] Leo Obrst: Concerning ONTOCOM, which is basically an ontology cost model, it identifies as

cost drivers: Building, Reuse, Personnel, and Project, with sub-categories for each of these.

Example: Building: Domain Analysis Complexity, Conceptualization Complexity, Implementation

Complexity, Instantiation Complexity, Required Reusability, Documentation Needs. Etc. It also has

developed a spreadsheet with these factors, and one can compute the estimated cost based on their

data and factor weights - http://ontocom.sti-innsbruck.at/.

[10:38] Steve Ray: I interpret John's presentation as addressing Capability as the metric in

question.

[10:40] Steve Ray: It's very hard to quantify Capability in the abstract, but in specific examples

they could be enumerated.

[10:40] Todd Schneider: John is providing a operational view.

[10:41] Rex Brooks: John's work is particularly useful in the SOA context, especially the emerging

ecosystem view.

[10:42] Rex Brooks: But the evaluation metrics need to be fleshed out a bit.

[10:42] Steve Ray: Slide 4 has some good raw material for metrics.

[10:43] Rex Brooks: It's difficult to measure inferencing except for accuracy, e.g. internal

consistency with its own definitions.

[10:44] Todd Schneider: Rex, the performance of inferencing should be measurable.

[10:44] Pat Barkman: Thanks John

[10:46] John Yanosy: You are welcome and sorry for not being able to provide more detail, but will be

posting more material on the wiki and think the previous metrics would be interesting to apply to

these business areas. Great job and will be more active in future. Good by thanks

[10:44] Rex Brooks: We'll get there eventually Todd,

[10:45] Bobbin Teegarden: Is part of the value of inferencing in code NOT written (and associated

costs')?

[10:46] Mary Balboni: Thanks Todd - :)

[10:47] Peter P. Yim: == Todd Schneider presenting ==

[10:50] Steve Ray: @Todd: Just because the consequence of error is different, it doesn't follow the

METRIC is different, just that the value and weighting of that metric may be different.

[10:52] Rex Brooks: There are some real misunderstandings about the differences between qualitative

and quantitative metric and the relationship between them, but that's almost a topic of its own. I

come from an Advertising background and we had to provide quant. to satisfy the customer's need to

rationalize while appealing to unstated qual. that are often the actual driving motivation to

purchase or not in the marketplace.

[10:52] Leo Obrst: As part of the DARPA HPKB (High Performance Knowledge Bases) and RKF (Rapid

Knowledge Formation) programs, large ontology integration efforts to solve a command and

control/situational awareness problem, the Program Manager Murray Burke (and before him, Dave

Gunning) tried to capture ontology axiom reuse metrics.

[10:54] Bobbin Teegarden: @Leo Results of that reuse work? Any references online?

[11:15] Leo Obrst: @Bobbin: yes, some of it is online. See the paper Schrag, Robert, Mike Pool, Vinay

Chaudhri, Robert C. Kahlert, Joshua Powers, Paul Cohen, Julie Fitzgerald, and Sunil Mishra.

Experimental Evaluation of Subject Matter Expert-oriented Knowledge Base Authoring Tools.

[10:56] Mike Bennett: Does this mean that it would be possible to define a quantitative difference

between ontologies with losts of "Equivalent class" links versus ontologies which make use of high

level patterns or archetypes? Could this make the case for better use of sharing and integration

ontologies? Just a thought.

[10:56] Steve Ray: OK. So far I have heard the following classes of metrics: Cost, Capability,

Performance, Quality, System Complexity. Any others?

[10:58] Steve Ray: People can un-mute themselves. *3 to un-mute; *2 to mute

[10:58] Mary Balboni: Maturity related to the depth of model and breadth of validation ...

[11:00] Steve Ray: @Mary: Wouldn't maturity manifest itself to a customer in terms of capability? In

other words, I would put the maturity of a model as a property that feeds in to the capability

metric.

[11:01] Steve Ray: @Mary: Put another way, I would imagine that a customer isn't as interested in the

maturity of an embedded ontology itself, but rather in how that might affect performance of their

system.

[11:01] Mary Balboni: @SteveRay - more capability may be part of maturity - or could be more

correctness while operational --- also had thoughts on Complexity can be measured like the old

Halsteads measures perhaps...

[11:02] Steve Ray: @Mary: OK. I accept that model maturity can manifest itself through several

classes of metrics, including correctness (which think of as a specialization of the Quality

metric).

[11:03] Mary Balboni: Operators and operands were counted and an assumption was made on how complex

code was based on those numbers - not sure if Ontology could be discected in such a way

[11:03] Peter P. Yim: unlike other maturity models, the Capability Maturity Model for Software

Engineering (SEI CMM) for example, in Ontology, "more mature" may correlate with "more

sophisticated, and better grasp of semantics (stronger semantics) in the system" and may not

correlate directly with performance, much less effectiveness, or even whether it is appropriate. ...

Therefore: (a) mapping metrics to the application framework may be necessary, (b) different metrics

should be expected at different level of Ontology Maturity (as how the the applications are

implemented would be radically different.)

[11:03] Steve Ray: I'm trying to separate in my own mind the distinction between intrinsic measures

of an ontology, versus extrinsic metrics of business value which will be the ones that a customer or

decision maker will be evaluating when being pitched.

[11:06] Rex Brooks: I'd be interested to hear how people think we can measure inferencing.

[11:06] Alex Mirzaoff: by success of the inference?

[11:07] Steve Ray: @Rex: Also by maximum compute time

[11:07] Rex Brooks: What happens if the inference extends over different systems that may offer

different definitions in different cases?

[11:08] Bobbin Teegarden: Measure value of inference in terms of equivalent code it would take to do

the same thing in code (as a 'savings' or negative cost)?

[11:09] Mary Balboni: cost avoidance

[11:09] Bobbin Teegarden: No, genuine 'savings'?

[11:09] Rex Brooks: @Bobbin: Excellent. Never thought of that. Point is, we need lots of use cases to

check against.

[11:11] Mike Bennett: On CMM, there is also a "Data Management Maturity Model" in development by the

EDM Council, in collaboration with Carnegie Mellon as owners of the CMM model. This is still in very

early stages of development, so there is potential to input to this with metrics for ontology

maturity if / when these are defined.

[11:11] Pat Barkman: but I think there are intrinsic benefits in using CMMi (MA/PPQA) in a

before/after comparision ... an existing implementation that gets an ontology added to the

architecture

[11:14] Rex Brooks: @Pat: Yes, seeing the difference in those before and after results would lead to

new insights, I'm sure.

[11:15] Michael Grüninger: Since we want to demonstrate the benefits of ontologies, let's first

consider the benefits in a software application. We can leverage the existing approaches to software

engineering by considering functional and nonfunctional requirements. For functional requirements,

we need to demonstrate that ontologies can be used to deliver new functionalities. For nonfunctional

requirements, we can use existing software metrics such as performance, cost, quality, maintenance.

In each case, we want to compare an application without an ontology and an application with an

ontology, and show that there is an improvement.

[11:11] Steve Ray: I completely concur with the points just made by MichaelGruninger.

[11:12] Peter P. Yim: +1

[11:13] Rex Brooks: +2

[11:17] Bobbin Teegarden: There must be some way to capture the wider comprehension, collaborative

common interactions, group understanding, ease of extension, ... some of the things that make use of

ontologies truly a step forward.

[11:17] Rex Brooks: == Leo presenting his maturity model ==

[11:17] Rex Brooks: Peter just put it up on the vnc!

[11:19] Terry Longstreth: Leo's is a very valuable start, but it doesn't address cost/value

[11:17] Mary Balboni: DoDAF also is expanding their framework to concentrate on data such as the

CMMI-DM - have not explored if DoDAF new version is addressing Ontology/semantics in detail

[11:18] Rex Brooks: @Mary: Slowly pulling teeth along the way--I participate in DoDAF Metamodel 2 WG.

[11:20] Mary Balboni: @Rex DoDAF is a late bloomer in Data Modeling .. :)

[11:20] Rex Brooks: @Mary: Yup!

[11:07] Pavithra Kenjige: How Are these different than system development life cycle?

[11:18] Michael Grüninger: On the other side, there are costs associated with using ontologies within

an application, and perhaps these are not completely covered by the analogy to software engineering

[11:20] Todd Schneider: Michael, one of those out-of-band costs/risks is the availability of

experienced people to do the work

[11:20] Alan Rector: For us, the notions of sustainability and persistence are more relevant than

maturity - or perhaps are some of the relevant metrics for maturity for ontologies.

[11:21] Michael Grüninger: Did the software engineering community ever do an analysis of the benefits

of using object-oriented approaches to software design? We are perhaps facing an analogous problem

...

[11:21] Steve Ray: @Todd: Your point raises the additional metric of Risk (in this case, risk of not

being able to maintain the system over time, for example).

[11:22] Jeff Zhuk: Mary, did you start using ontology or in the development stage?

[11:23] Alan Rector: One key step for our cases is when the primary identifiers ontology moves from

text identifiers that inevitably change and are language specific to "nonsemantic IDs", and when

there is a sensible ID management scheme in place with the display names in annotations (usually

rdf:label or one of the skos:label family.

[11:24] Mary Balboni: @MichaelGruningerThere have been OO studies in SW Development - Rationale may

have study papers .. probably biased .. but the OO giants in industry are at Rationale

[11:25] Mary Balboni: @Yefim Jeff Not at the moment implementing an Ontology but interested in its

usefulness in an IA domain

[11:26] Alan Rector: Maturity of "ontology technology" rather than of a specific ontology?

[11:28] Mike Bennett: Mills did mention one or two case studies where there were measured costs of

doing it the hard way, and then using ontologies. However most of our Case Studies to date have not

included explicit metrics.

[11:33] Peter P. Yim: folks from Canada did have some hard numbers - ref.

http://ontolog.cim3.net/forum/ontology-summit/2011-02/msg00017.html & 2nd presentation at:

http://ontolog.cim3.net/forum/ontology-summit/2010-12/msg00002.html

[11:30] Steve Ray: Probably about time to wrap things up...

[11:31] Jeff Zhuk: Thanks!

[11:31] Pat Barkman: Productive and informative ... great session, thanks all

[11:31] Steve Ray: Thanks for a good session!

[11:31] Mary Balboni: thanks to all!

[11:33] Peter P. Yim: Great session ... thank you!

[11:33] Peter P. Yim: -- session ended: 11:32am PST --

-- end of in-session chat-transcript --

  • Further Question & Remarks - please post them to the [ ontology-summit ] listserv
    • all subscribers to the previous summit discussion, and all who responded to today's call will automatically be subscribed to the [ ontology-summit ] listserv
    • if you are already subscribed, post to <ontology-summit [at] ontolog.cim3.net>
    • (if you are not yet subscribed) you may subscribe yourself to the [ ontology-summit ] listserv, by sending a blank email to <ontology-summit-join [at] ontolog.cim3.net> from your subscribing email address, and then follow the instructions you receive back from the mailing list system.
      • please email <peter.yim@cim3.com> if you have any question.

Audio Recording of this Session

  • To download the recording of the session, click here
    • the playback of the audio files require the proper setup, and an MP3 compatible player on your computer.
  • Conference Date and Time: 17-Feb-2011 9:40am~10:32am PST
  • Duration of Recording: 1 Hour 43 Minutes
  • Recording File Size: 11.8 MB (in mp3 format)
  • suggestions:
    • its best that you listen to the session while having the respective presentations opened in front of you. You'll be prompted to advance slides by the speaker.
    • Take a look, also, at the rich body of knowledge that this community has built together, over the years, by going through the archives of noteworthy past Ontolog events. (References on how to subscribe to our podcast can also be found there.)

Additional Resources

  • Homepage of OntologySummit2011
    • Ontology Summit 2011 Launch Event - ConferenceCall_2011_01_20
    • "Ontology Application Framework" session - ConferenceCall_2011_02_03
    • "Ontology Applications & Use Cases - I" - ConferenceCall_2011_01_27
    • "Ontology Applications & Use Cases - II" - ConferenceCall_2011_02_10
  • [ontology-summit] mailing list archives - http://ontolog.cim3.net/forum/ontology-summit/
  • Homepage of this annual Summit series - see: OntologySummit

For the record ...

How To Join (while the session is in progress)

Conference Call Details

  • Date: Thursday 17-Feb-2011
  • Start Time: 9:30am PST / 12:30pm EST / 6:30pm CET / 5:30pm GMT / 17:30 UTC
    • see world clock for other time zones
    • do note that this is 1 hour earlier than the usual 1:30 EST Ontolog event time!
  • Expected Call Duration: ~2.0 hours
  • Dial-in Number:
    • from a US telephone (US): +1-218-844-8060 (domestic long distance cost will apply)
    • When calling in from a phone, use Conference ID: "4389979#"
    • from Europe, call:
      • Austria 0820-4000-1577
      • Belgium 070-35-9992
      • France 0826-100-280
      • Germany 01805-00-7642
      • Ireland 0818-270-037
      • Italy 848-390-179
      • Spain 0902-886-056
      • Switzerland 0848-560-327
      • UK 0844-581-9148
    • callers from other countries please dial into either one of the US or European numbers
  • Shared-screen support (VNC session), if applicable, will be started 5 minutes before the call at: http://vnc2.cim3.net:5800/
    • view-only password: "ontolog"
    • if you plan to be logging into this shared-screen option (which the speaker may be navigating), and you are not familiar with the process, please try to call in 5 minutes before the start of the session so that we can work out the connection logistics. Help on this will generally not be available once the presentation starts.
    • people behind corporate firewalls may have difficulty accessing this. If that is the case, please download the slides above and running them locally. The speaker(s) will prompt you to advance the slides during the talk.
  • Discussions and Q & A:
    • (Unless the conference host has already muted everyone) Please mute your phone, by pressing "*2" on your phone keypad, when a presentation is in progress. To un-mute, press "*3"
    • You can type in your questions or comments through the browser based chat session by:
    • or point your browser to: http://webconf.soaphub.org/conf/room/ontolog_20110217
      • instructions: once you got access to the page, click on the "settings" button, and identify yourself (by modifying the Name field). You can indicate that you want to ask a question verbally by clicking on the "hand" button, and wait for the moderator to call on you; or, type and send your question into the chat window at the bottom of the screen.
    • (when everyone is muted) If you want to speak or have questions or remarks to make, please "raise your hand (virtually)" by click on the "hand button" (lower right) on the chat session page. You may speak when acknowledged by the speaker or the session moderator (again, press "*3" on your phone to unmute). Test your voice and introduce yourself first before proceeding with your remarks, please. (Please remember to click on the "hand button" again (to lower your hand) and press "*2" on your phone to mute yourself after you are done speaking.)
    • thanks to the soaphub.org folks, one can now use a jabber/xmpp client (e.g. gtalk) to join this chatroom. Just add the room as a buddy - (in our case here) ontolog_20110217@soaphub.org ... Handy for mobile devices!
  • Please note that this session will be recorded, and the audio archive is expected to be made available as open content to our community membership and the public at-large under our prevailing open IPR policy.

Attendees

  • Expecting:
    • ... if you are coming to the session, please add your name above (plus your affiliation, if you aren't already a member of the community) above; or e-mail <peter.yim@cim3.com> so that we can reserve enough resources to support everyone's participation. ...



Retrieved from "https://ontologforum.com/index.php?title=ConferenceCall_2011_02_17&oldid=328"