Framework for evaluating communication – ASTC 2017

I’m attending the Technical Communicators Conference 2017, the annual conference of the Australian Society for Technical Communication (ASTC). This post is a live blog of a session at the conference. In other words, I’m writing as the presenter speaks. Any mistakes are mine, and all credit goes to the presenter.

Neil James presented a session titled Will it work? A new integrated framework for evaluating communication. Neil is the executive director of the Plain English Foundation.

The first question that Neil addressed was, “Why do people write ‘Manglish’, or mangled English?” This topic wasn’t a prepared part of his presentation. Instead, it was something the conference organiser asked Neil to consider, following on from a session yesterday. Here are some of the contributing factors that Neil mentioned:

• At school, we imbibe the notion that complex writing is better writing. Waffle gets reasonable marks, provided it’s elegant waffle.
• Early in our careers in the professional and technical workplace, mastering and using the technical jargon of our field gives us a stronger feeling of belonging.
• When we learn the tech vocabulary of a particular industry, it’s difficult to adjust to communicating with a lay audience.
• Institutional culture reinforces the language patterns. Large organisations move slowly. It’s hard to change their processes. When you do successfully introduce change, the organisation moves steadily along the new path.
• Language is used as an expression of power. Sometimes, people deliberately use jargon to protect their financial interests or to manipulate public opinion. An example is from the airline industry, when people use the term “loss of separation” of two planes, which means the two planes collided.

Evaluating technical communication: Testing versus techniques

Neil talked about the various ways of evaluating technical documentation. There tend to be two groups of people: those who focus on user testing, and those who focus on techniques such as readability formulas.

Communication professionals tend to evaluate their work using a technique they learned near the beginning of their careers, and to use a fairly narrow range of techniques. Neil advocates making wider use of the spectrum of methods available.

Neil’s talk covered:

• The evaluation methods available
• Criteria for selecting the right method
• Examples

The spectrum of evaluation methods

One of the key things that’s happening to our role is that we need to produce a wider variety of technical communication. Neil says we also need to consider a wider range of evaluation methods and tools.

The methods for evaluating tech communication tend to focus on the following three areas, which Neil talked through in detail:

• Focus on text – tool based or expert review
• Focus on users – preference testing or performance testing
• Focus on outcomes – transactions (such as errors, phone calls, revenue) or attitudes and behaviours

Note that you can blend and scale these techniques to suit your environment and requirements. The various methods are not as difficult as we sometimes think they are, and learning about them adds to our professional expertise.

Criteria for selecting the right method

Many of the criteria are familiar to communication professionals. The value of the framework is that it brings these criteria together in a structured way.

Here’s a summary of the factors that affect your choice of method:

• Requirements related to the reader:
  • Number of readers, literacy skills, knowledge of content.
  • Complexity of the content and of the task.
  • Urgency of the task.
• Requirements of the business:
  • Timing (standard to disruptive).
  • Costs.
  • Risks.
• Outcomes shared by both: Purpose and impact of the communication. Sometimes the purpose of the communication may be perceived differently by the organisation and the reader. Similarly for the impact.

Putting it all together

Neil showed us a tabular representation of the above items, which you can use to decide which types of evaluation to use. He then ran through some examples showing where and how he’d used the framework to decide which types of evaluation would be useful for some specific use cases.

The framework is still in its early days. Neil and his colleague, Susan Kleimann, are still testing the framework and getting feedback on it from customers. One point he emphasised is that the framework helps to secure budget for the evaluation part of a project. Once you put the framework in front of customers, they can see the need for the evaluation phase.

To get a more visual idea of the framework, take a look at an early version of the slides from a conference in Dublin in 2015. Neil confirmed that the information on those slides is essentially the same as presented today.

Thanks Neil for introducing a useful framework.