planning community to create and improve tools
and techniques for supporting the main design phases of a planning domain model; and ( 3) to provide
new interesting and challenging models that can be
used for testing the performance of state-of-the-art
planning engines. In order to achieve the mentioned
aims, ICKEPS 2016 focused on on-site modeling of
challenging scenarios, performed by small teams.
This article summarizes the ICKEPS held in 2016.
More information about the competition, including
complete scenario descriptions, can be found on the
ICKEPS 2016 website. 1
Format and Participants
ICKEPS 2016 format included two main stages: On-site modeling and demonstration.
During the on-site modeling stage, each team
received a set of scenarios description and had to
exploit the available time for generating the corresponding models. Four scenarios were provided. Two
of them — Star Trek, Rescue of Levaq, and Roundabout — required temporal constraints, while the
other two — RPG and Match-Three, Harry! — only
required classical reasoning. Participants were free to
select the scenarios to tackle and had no restrictions
on the number and type of tools that can be used.
The only constraints were on the available time (six
hours were given) and on the maximum size of teams
(at most four members).
The day after the on-site modeling, each team had
10 minutes to present and demonstrate the aspects
of the knowledge engineering process they exploited
for encoding the scenarios. Specifically, teams were
expected to discuss the division of work among team
members, the tools used, key decisions taken during
the encoding, and the issues they faced.
Teams were then ranked by a board of judges,
which included Minh Do (NASA, USA), Simone Frati-ni (ESA, Germany), Ron Petrick (Heriot-Watt University, UK), Patricia Riddle (University of Auckland,
New Zealand), and David Smith (NASA, USA). The
evaluation process will be described in the corresponding section below. Noteworthy, judges were
presented during the demonstrations session and
had the opportunity to ask questions and discuss relevant aspects of the knowledge engineering process
the teams followed.
The competition had two tracks: the PDDL track,
where teams had to generate PDDL models using
PDDL features up to those introduced in version 3. 1,
and the Open track, where teams could encode models in any other language. However, for the open
track, participants were also required to provide a
planner able to deal with the selected language. Sixteen people, divided into six teams, took part in the
competition. One team entered the Open track,
while the remaining five decided to participate in the
Participants came from institutions in Australia,
Brazil, Canada, USA, Japan, and the United Kingdom.
The level of expertise of participants covered various
academic ranks, that is, Ph.D. students, lecturers,
research fellows, and professors. One team was composed only of industry experts.
The board of judges evaluated each team by consid-
ering two main aspects: the exploited knowledge
engineering process and the quality of the generated
The knowledge engineering process was assessed
once for each team, regardless of the number of scenarios the team was able to encode. Three main criteria were taken into account: teamwork, method,
and tools. Teamwork focused on the degree of cooperation and effective collaboration among team
members. In terms of the method, effectiveness and
systematicity of the knowledge engineering process
were assessed. Finally, the innovation and originality
of exploited tools, and their actual usefulness (that is,
the support their use provided to the process) were
To assess the quality of the generated models, the
organizers provided the judges with the models the
teams had submitted along with quantitative and
qualitative information about these models. Qualitative information included evaluations about correctness, (that is, whether all the requirements were correctly handled); readability (how easy it was to read
and understand the model); generality (if the domain
model could be reused on different problem
instances); and originality, where the use of innovative ways for modeling element or interactions was
evaluated. Quantitative information included statistics on the number of types, number of predicates,
number of operators, total number of lines, and the
average (maximum) number of parameters, effects,
and preconditions per operator. Moreover, in the
PDDL track, the run time and quality of solutions
generated by 10 well-known planners ( 5 classical and
5 temporal) were provided to judges. For teams participating in the Open track, the corresponding performance of the planner(s) submitted by the participants were provided to judges.
In accordance with the aims of the competition,
emphasis was given to good practice in knowledge
engineering, with particular regard to the degree of
cooperation between the members of each team. For
this reason the judges used a 0– 100 scale, where up to
45 points could be awarded for the knowledge engineering process, and the remaining 55 points could
be assigned according to the number and quality of
generated models, as follows: Star Trek, the Rescue of
Levaq (up to 20 points); Roundabout (up to 15
points); Match-Three, Harry! (up to 10 points); and
RPG (up to 10 points).