Difference between revisions of "Ontology Metrics"

From NCBO Wiki
Jump to: navigation, search
(REST service access)
Line 1: Line 1:
 
= Ontology Metrics in BioPortal =  
 
= Ontology Metrics in BioPortal =  
  
This page describes the metrics that BioPortal calculates for the ontologies in its repository. In general, BioPortal will calculate the metrics when the ontology is uploaded and will store it as part of  the [http://www.bioontology.org/wiki/index.php/BioPortal_Metadata Ontology Metadata]. There are two groups of metrics:
+
This page describes the metrics that BioPortal calculates for the ontologies in its repository. BioPortal calculates metrics when the ontology is uploaded and will store them as part of  the [http://www.bioontology.org/wiki/index.php/BioPortal_Metadata Ontology Metadata]. There are two groups of metrics:
 
* '''Statistical''' metrics
 
* '''Statistical''' metrics
 
* '''Quality-control''' and '''quality-assurance''' metrics
 
* '''Quality-control''' and '''quality-assurance''' metrics
  
Each ontology may have all, some, or no values filled in for its metrics. Some metrics are meaningful only for ontologies in a specific representation language (e.g., number of axioms matters only for OWL ontologies; there are no individuals to count in the ontologies in OBO format).  
+
Some metrics are meaningful only for ontologies in a specific representation language (e.g., there are no individuals to count in the ontologies in OBO format).  
  
 
== Statistical Metrics ==  
 
== Statistical Metrics ==  
 
We calculate the following statistics on each ontology in BioPortal:
 
We calculate the following statistics on each ontology in BioPortal:
  
* '''number of classes''': the number of classes in the ontology. For OWL ontologies
+
* '''number of classes''': the number of named (not anonymous) classes in the ontology.
** does not include imported classes
+
** includes only named classes (no anonymous classes in this count)
+
 
* '''number of properties:''' the number of properties or slots in the ontology.
 
* '''number of properties:''' the number of properties or slots in the ontology.
* '''number of individuals:''' the number of individuals in the ontology. Note:
+
* '''number of individuals:''' the number of individuals in the ontology.  
** Individuals exist only in OWL and Protege ontologies.
+
** An individual in OWL is a resource that is neither a class, nor a property.
+
* '''number of axioms:''' the number of axioms in an OWL ontology.
+
 
* '''maximum depth:''' the maximum depth of the hierarchy tree.  
 
* '''maximum depth:''' the maximum depth of the hierarchy tree.  
 
** For OWL, RDFS, Protege, and RRF ontologies, we consider only the ''is-a'' relationship as a hierarchical relationship
 
** For OWL, RDFS, Protege, and RRF ontologies, we consider only the ''is-a'' relationship as a hierarchical relationship
Line 28: Line 23:
  
 
* '''classes with only one subclass''': a list of classes that have only one subclass in the ''is-a'' hierarchy. While technically there is no problem in having only one subclass, this situation often indicates that either the hierarchy is under-specified, or the distinction between the class and the subclass is not appropriate.
 
* '''classes with only one subclass''': a list of classes that have only one subclass in the ''is-a'' hierarchy. While technically there is no problem in having only one subclass, this situation often indicates that either the hierarchy is under-specified, or the distinction between the class and the subclass is not appropriate.
* '''classes with too many subclasses''': a list of classes that have more direct subclasses than a certain threshold. The specific threshold is an internal parameter in BioPortal (currently set to 25). This situation may be an indication that additional distinctions and categorization are needed.
+
* '''classes with more than 25 subclasses''': a list of classes that have more than direct subclasses. A class that has more than 25 subclasses is a candidate for additional distinctions and categorization are needed.
* '''classes with no definition''': a list of classes that have no value for the definition property. For ontologies in OBO and RRF formats, the property for definition is part of the language. For OWL ontologies, the authors specify this property as part of the ontology metadata (the default is <tt>skos:definition</tt>). If the ontology author has not specified any property as a definition property, we do not fill out this value.
+
* '''classes with no definition''': a list of classes that have no value for the definition property. For ontologies in OBO and RRF formats, the property for definition is part of the language. For OWL ontologies, the authors specify this property as part of the ontology metadata (the default is <tt>skos:definition</tt>).  
* '''univocality''' (future implementation): the consistency of expression of concepts in an ontology (developed by Karin Verspoor∗, Daniel Dvorkin, K. Bretonnel Cohen, and Lawrence E. Hunter, University of Colorado)
+
 
+
== Auditing and other information ==
+
* '''auditing information''': a set of <author, classes> pairs that indicates which lists the classes that each author contributed to the ontology. We use the property that the ontology developers specify in their metadata to determine who is the author of the class. If the ontology author has not specified any property as a definition property, we do not fill out this value.
+
 
+
  
 
Please, contact us at [mailto:support@bioontology.org support@bioontology.org], if there are additional metrics of any kind that you would like to see.
 
Please, contact us at [mailto:support@bioontology.org support@bioontology.org], if there are additional metrics of any kind that you would like to see.
Line 41: Line 31:
 
= Generating metrics =  
 
= Generating metrics =  
  
At the moment, we ''do not'' generate metrics as part of the ontology load process, but we will shortly. The process will be as follows: When BioPortal loads and parses a version of an ontology, it then starts a process that calculates the metrics and stores the values as part of the metadata about that ontology version.
+
When BioPortal loads and parses a version of an ontology, it then starts a process that calculates the metrics and stores the values as part of the metadata about that ontology version.
  
 
== Metrics computation and ontology metadata ==
 
== Metrics computation and ontology metadata ==

Revision as of 14:05, 2 September 2011

Contents

Ontology Metrics in BioPortal

This page describes the metrics that BioPortal calculates for the ontologies in its repository. BioPortal calculates metrics when the ontology is uploaded and will store them as part of the Ontology Metadata. There are two groups of metrics:

  • Statistical metrics
  • Quality-control and quality-assurance metrics

Some metrics are meaningful only for ontologies in a specific representation language (e.g., there are no individuals to count in the ontologies in OBO format).

Statistical Metrics

We calculate the following statistics on each ontology in BioPortal:

  • number of classes: the number of named (not anonymous) classes in the ontology.
  • number of properties: the number of properties or slots in the ontology.
  • number of individuals: the number of individuals in the ontology.
  • maximum depth: the maximum depth of the hierarchy tree.
    • For OWL, RDFS, Protege, and RRF ontologies, we consider only the is-a relationship as a hierarchical relationship
    • For OBO format ontologies, we consider the following relationships as hierarchical relationships: is-a, has-part, inverse of develops-from
  • average number of siblings: the average number of siblings at one level in the tree.
  • maximum number of siblings: the maximum number of siblings in the ontology.

Quality-Control and Quality-Assurance Metrics

We calculate the following metrics that may give some indication of the quality of the ontology and help ontology authors improve the quality. To improve their usefulness, we limit the size of lists for any one of these metrics to 200. Ontologies with more than 200 classes in a category will still have the total number of classes in that category counted, but no list will be available.

  • classes with only one subclass: a list of classes that have only one subclass in the is-a hierarchy. While technically there is no problem in having only one subclass, this situation often indicates that either the hierarchy is under-specified, or the distinction between the class and the subclass is not appropriate.
  • classes with more than 25 subclasses: a list of classes that have more than direct subclasses. A class that has more than 25 subclasses is a candidate for additional distinctions and categorization are needed.
  • classes with no definition: a list of classes that have no value for the definition property. For ontologies in OBO and RRF formats, the property for definition is part of the language. For OWL ontologies, the authors specify this property as part of the ontology metadata (the default is skos:definition).

Please, contact us at support@bioontology.org, if there are additional metrics of any kind that you would like to see.


Generating metrics

When BioPortal loads and parses a version of an ontology, it then starts a process that calculates the metrics and stores the values as part of the metadata about that ontology version.

Metrics computation and ontology metadata

Metric generation relies on the following information being specified in the ontology metadata:

For OBO and RRF ontologies:

  • property used to specify the authors of a class

For OWL ontologies:

  • property used to specify preferred name (default rdfs:label, and if not specified, then rdf:ID)
  • property used to specify synonyms (default skos:altLabel)
  • property used to specify the definition of a class (default skos:definition)
  • property used to specify the author of a class (default dc:creator)

Representing metrics in BioPortal

The metrics in BioPortal are stored as part of the ontology metadata. Specifically, each instance of the OMV:Ontology class, which describes the metadata about a single ontology version, has a number of properties that store the metrics. Some of these properties are imported from the Ontology Metadata Vocabulary, OMV (the omv: prefix) and the rest are defined locally in the BioPortal Metadata Ontology (the metrics: prefix). These properties include:

  • OMV:numberOfClasses
  • OMV:numberOfProperties
  • OMV:numberOfAxioms
  • OMV:numberOfIndividuals
  • metrics:averageNumberOfSiblings
  • metrics:maximumNumberOfSiblings
  • metrics:classesWithNoDocumentation
  • metrics:maximumDepth
  • metrics:preferredMaximumSubclassLimit
  • metrics:classesWithSingleSubclass
  • metrics:classesWithNoAuthor
  • metrics:classesWithMoreThanXSubclasses
  • metrics:classesWithMoreThanOnePropertyValueForPropertyWithUniqueValue

Accessing metrics in BioPortal

BioPortal users will be able to access ontology metrics in two ways:

  • through the BioPortal user interface, as part of the Ontology Details tab (not implemented yet)
  • through a dedicated REST service that returns a MetricsBean

REST service access

Applications can use the following REST service to access metrics information for a specific ontology version. Note: there is no version of this service that uses virtual ontology id yet.

The service returns a ontologyMetricsBean that contains the version id for the ontology and the values for the metrics that have values computed for them:

<success>
  <accessedResource>/bioportal/ontologies/metrics/42031</accessedResource>
  <accessDate>2010-01-25 15:15:36.742 PST</accessDate>
  <data>
    <ontologyMetricsBean>
      <id>42031</id>
      <numberOfAxioms>0</numberOfAxioms>

      <numberOfClasses>293</numberOfClasses>
      <numberOfIndividuals>0</numberOfIndividuals>
      <numberOfProperties>8</numberOfProperties>
      <maximumDepth>6</maximumDepth>
      <maximumNumberOfSiblings>19</maximumNumberOfSiblings>
      <averageNumberOfSiblings>1</averageNumberOfSiblings>

      <classesWithOneSubclass>
        <string>APO:0000003</string>
        <string>APO:0000039</string>
        <string>APO:0000064</string>
        <string>APO:0000084</string>
        <string>APO:0000101</string>

        <string>APO:0000105</string>
        <string>APO:0000210</string>
        <string>APO:0000213</string>
        <string>APO:0000215</string>
        <string>APO:0000255</string>
      </classesWithOneSubclass>

      <classesWithMoreThanXSubclasses/>
      <classesWithNoDocumentation/>
      <classesWithNoAuthor/>
      <classesWithMoreThanOnePropertyValue/>
    </ontologyMetricsBean>
  </data>
</success>

If a given ontology has more than 200 classes identified for any single metric, then that metric will provide a parsable error message in place of the class list. Two statuses are available:

  • limitpassed: signifies that the 200 class limit was reached and passed, presents this error followed by the total number of classes in the ontology that matches this metric. limitpassed can be presented in two ways:
    • <string>limitpassed:NNNN</string>
    • <string>limitpassed:</string>
      <string>NNNN</string>
  • alltriggered: Every class in the ontology matched this metric. alltriggered is presented like this:
    • <string>alltriggered</string>


<success>
  <accessedResource>/bioportal/ontologies/metrics/40644</accessedResource>
  <accessDate>2010-01-25 15:13:06.40 PST</accessDate>
  <data>
    <ontologyMetricsBean>
      <id>40644</id>
      <numberOfClasses>74646</numberOfClasses>
      <numberOfIndividuals>0</numberOfIndividuals>
      <numberOfProperties>191</numberOfProperties>
      <maximumDepth>14</maximumDepth>
      <maximumNumberOfSiblings>3215</maximumNumberOfSiblings>
      <averageNumberOfSiblings>139</averageNumberOfSiblings>
      <classesWithOneSubclass>
        <string>limitpassed:5153</string>
      </classesWithOneSubclass>
      <classesWithMoreThanXSubclasses>
        <entry>
          <string>limitpassed:</string>
          <string>22932</string>
        </entry>
      </classesWithMoreThanXSubclasses>
      <classesWithNoDocumentation>
        <string>limitpassed:37788</string>
      </classesWithNoDocumentation>
      <classesWithNoAuthor>
        <string>alltriggered</string>
      </classesWithNoAuthor>
      <classesWithMoreThanOnePropertyValue/>
    </ontologyMetricsBean>
  </data>
</success>

Personal tools
Namespaces

Variants
Actions
Navigation
Toolbox