ONT2 with nominals rolled back to concepts. The
first four cases were created for performance testing,
The fifth one was the result of performance tuning.
We tested three reasoners (FacT++ v1.6.3, Pellet
v2.2.0, and HermiT v1.3.8.) on the five ontologies
using Protégé v4.3.0 on Intel i7-4770 with 16 GB
RAM running 64-bit Ubuntu 12.04. The execution
times are given in table 4. We make the following
observations: ( 1) Of the reasoners, FacT++ has the
best overall performance, followed by HermiT and
Pellet. ( 2) Of the ontologies, LEX1 has the best overall performance, it has a 1: 21 class to individual ratio;
and ONT1 has good overall performance and has no
individuals. ( 3) The performance, though within
acceptable limits, begins to degrade for ONT2 and
ONT3. Hermi T and Pellet are up to two orders of magnitude slower than Fac T++ for these ontologies.
To understand where the reasoner was spending
time, we profiled ONT3 using Pellet3 and computed
the classification time for each concept. Using this, a
Pareto chart was prepared; see figure 8. Observe that
96 percent of the reasoner’s time is spent in classifying 20 percent of the terms.
We analyzed these terms and found that most of
these had owl:hasValue restriction in its definition.
To verify the impact of owl:hasValue on performance,
we created ONT4 from ONT2 by changing fillers of
owl:hasValue into concepts and rewriting owl:hasValue as existential restriction. Now, ONT4
outperforms ONT2 and ONT3, and has a comparable performance to ONT1 (table 4).
From this we conclude that creation of individuals
has less impact on performance, as seen in LEX1, but
using them in owl:hasValue restriction degrades performance, as seen in ONT2, ONT3. This is true for Hermi T and Pellet. In our test, FacT++ consistently outperforms HermiT and Pellet, and for our ontology
Fac T++ is unaffected by nominals.
This performance test is solely based on execution
time. We did not compare the inferences from these
reasoners, so we do not know if there is any qualitative difference in the inferences from these reasoners.
Deployment and Maintenance
We (Ford) verified the completeness of the new OWL
ontology by developing a tool to compare it to the
KL-ONE version. The delivered OWL ontology needed to be validated and verified as the first step toward
deployment. This process consisted of several steps.
Initially, the OWL ontology was loaded into an Allegrograph server and we wrote various SPARQL queries
to determine if the results returned were as expected.
In cases where the results were not satisfactory, we
then examined the ontology and made modifications
if they were required. This manual validation went
on for a period of several weeks until we were certain
that the OWL ontology was complete and usable.
The next phase of the validation process utilized
Table 3. Ontology Test Cases.
Case Individuals Individuals Classes
LEX1 lex leaves 6,780 317
ONT1none 0 12,815
ONT2 lex leaves 5,679 7,136
ONT3 lex and obj leaves 6,898 5,917
ONT4 lex leaves minus nominals 5,136 7,679
Table 4. Classification Time (in Seconds).
In DL profile, AL stands for attributive language, E for existential restriction,
H for subrole, and O for nominals.
Language Pro;le Classi;cation Time
Case OWL DL FacT++ HermiT Pellet
LEX1 OWL 2 EL AL 0.2 0.8 0.7
ONT1 OWL 2 EL ALEH 1. 6 12 4
ONT2 OWL2EL ALEHO 2. 3 74 564
ONT3 OWL2EL ALEHO 2. 7 352 716
ONT4 OWL 2 EL ALEH 1. 7 13 4
20 40 60 80
Shows the classi;cation time
for top-k terms. The top 10%
takes 65% of the total time,
and the top 20% takes 96% of
the total time.
Cumulative Term Count (%)
Figure 8. Pareto Chart. Time Versus Terms for ONT3.