SUO: Thoughts on the ontology evaluation questions

[Allan Terry <aterry@teknowledge.com>]

Title:

Folks
  Some comments on the evaluation questions.

1. Maturity
(How ready is it to use now? What capabilities have already been demonstrated?
Time  and resources needed to start using? Potential for improvement.)

We should separate the maturity of the ontology from that of its associated toolset.  For the ontology, maturity is partly the degree in which it meets its stated development goals and the quality with which it does so  We could also consider the completeness of the ontology with respect to the target domain.  Needless to say, drawing that boundary is non-trivial, and for an upper ontology the lower bound is somewhat arbitrary. It is probably obvious to this group but I'll say it anyway.  Size is a not terribly relevant indicator of "goodness"; better to think in terms of  completeness for a purpose and quality.   Longevity is a heuristic indicator only--an ontology that continuously develops over a long period can improve or can become as much of an accretionary mess as a long-lived program can.  A final dimension of maturity might be the degree to which the ontology recognizes and handles issues within its scope.  An ontology of process had better have a way to deal with the flow of time and probably needs to be able to represent situations that have changed without running into contradiction.

2. Robustness
(Heavy weight vs. light weight ontology features? Potential for improving  robustness?  How well will it handle known requirements, such as those listed in SUO Scope and  Purpose.)

What is robustness of an ontology vs of its tools?  In software, robustness refers to the ability of a program to tolerate errors, disruptions, and unexpected inputs. This is accomplished by redundancy, error checking at runtime, and QA during development.   In ontology, it might mean ability to express unexpected concepts.  For example, you have an ontology for the transportation domain and discover it is also useful for discussing network traffic.  I don’t think redundancy is generally a good thing in an ontology, there should not be many ways to express a single concept.  One could describe a QA process for an ontology but it is an open research question how to determine if an ontology is “correct”.

There is also robustness in the sense of scalability. That seems like a tools issue.  There is nothing in logic or KIF that prohibits an ontology of 50m axioms.  But nobody will edit it or reason with it using today's tools

4. Language Flexibility
(What ontology language is it in? How stable is language?
If desired,  could it be written in a different ontology language?)

This sounds like a question of expressivity on the theoretical side and ease of translation on practical side.  An ontology that is  purely a term hierarchy with a small number of relations cannot be used to solve problems requiring full first order logic. Any notation is theoretically  translatable into any other of  equal expressivity, but the tool set  may be lacking.  A graph representation could be deemed more intuitive but then have problems scaling up in size/complexity of the ontology.

6. Domain Friendly
(How easy to develop domain ontologies based on upper ontology?)
Just a little squishy.  If you use time for creation of the domain ontology as a metric, that depends on the size of the domain and its novelty.  I am not sure how happy I would be with "finished axioms/hour".  And if it is a factual ontology the norms are going to be different.  If we take concept reuse as a metric, that may be more interesting

-- Allan Terry