Paper introducing the methodology of the similarity index in field studies

Early in my post-doc project, our research group was contacted by an emergency control centre in a major oil company. The centre handles incidents on offshore oil and gas rigs, events like fires or gas leaks, heavy weather or ships on collision course. The centre's task is to construct a coherent image of the emergency based on the incoming information, and to advise and organize the different resources involved in handling the emergency. The centre wanted our help to identify areas for further improving their emergency handling.

My initial approach to the request was to develop measures for individual and team mental representation. That is, does each team member have a good idea of what's going on at the oil rig, and does the team as a whole have a good idea of this? However, it quickly became apparent that since both actual events and training exercises in this setting are non-transparent, dynamic and interactive, it would be challenging to establish a ground truth for what the different team members should be expected to know at different times through an exercise.

We arranged a series of unstructured and semistructured training exercises for all the emergency teams in the organization. At planned intervals we "froze" the exercise, and had all team members answer questions about the situation and the team's work. But the challenge remained of how to assess the responses of these questions. The experts were reluctant to say that a given team member should hold a given bit of information about the situation at a given point in time.

The solution I chose was to compare the answers between the team members. I developed two algorithms that were applicable to all the questions. The first of these returned a value from 0 to 1, indicating how similar your answer was to the rest of the team. The second returned a value from 0 to 1, indicating how similar your answer was to the team leader. I argued that the first calculation should approach a measure of shared mental models, that is, the extent to which the team member had the same representation of the incident as the rest of the team. The second calculation should approach a measure of situation awareness, as a higher score should be associated with more accurate knowledge, given the assumption that the team leader was the best informed member of the team.

As this was a fairly (though not completely) novel methodological approach, before going any further we published a paper that went into some detail on describing the measures and the calculation of the scores. This was published in Journal of Cognitive Engineering and Decision Making: The similarity index as an indicator of SMM and SA in field studies. This paper did not test any research hypotheses, as this was reserved for future papers.