How analytics can change your world, at STC Summit
This week I’m attending STC Summit 2019, the annual conference of the Society for Technical Communication (STC). I’m blogging my notes from the sessions that I attend. Thanks and all credit go to the speakers. Any mistakes are my own.
Analytics! They’re all the rage, and something I’m very interested in. David Payne presented a session called Analytics Can Change Your World. The subtitle of the talk was Or Justify Your Existence.
David’s goal in this presentation is to help us find the data that will help us prove our worth within an organisation. He said that the tool you use is not important. It’s what you do with it that matters.
The story of a team’s journey into analytics
David told the story of his team, which started from a small size with little clout. The team also had no insight into how their content was used or how often downloads occurred. They didn’t know who used the docs.
They started by collecting data using surveys. They had no way of finding out where users started from. The survey consisted of 20 questions. They had 300 responses over 3 years. The results were largely that the docs were “the worst ever seen”, “horrible”, and so on. There was no actionable feedback.
Next they added feedback links to the doc pages. Received thousands of responses. Not one of them was about the docs themselves.
The team also now knows how many people download the docs. The number has increased from 4 downloads in 2013 to more than 60,000 in 2017.
What to do if no-one is reading the online help?
At first, page views for the online help were very low.
- Advertise – for example, add a banner on the printed docs.
- Encourage add a hook in the product itself (Clippy?) to encourage people to view the online help.
- Engage – talk to customers about how people use the online help.
David showed how the page views for the online help increased year by year.
Using your metrics
When you have the data, you need to start asking questions. David walked through these:
- Which OS and browser are they using? David mentioned that his business stakeholders wanted this metric, even though to him it’s not relevant. Lesson: Have the pieces of data you want, but be ready to answer all sorts of questions.
- Did the searches yield the results that the user wanted? For example, some information is legitimately not in your help because it’s relevant to other organisations, but people still search for it. In David’s example, over 60% of search results were not the ones the customer wanted. Tactic: David ran focus groups with customers. Many of the customers said that they turned off the online help for their users. The reason was that the product was very configurable, and thus the screenshots and information in the help were for the baseline system, which was not useful in a highly configured system.
- How many page views per product? This result was not very useful, as it showed nearly 50% under the category “other” – these relate to search results and the home page, rather than single pages. This result did show, however, that there were 2000 topics in the docs. That’s too many for people to navigate around. David’s team is now working on reducing the number of topics and on improving context sensitivity.
- How effective is context sensitivity? See how often people are arriving on the first page of the help as opposed to specific pages.
Some strategies are to examine the data:
- Focus on improving the high-traffic pages. Create a home page that highlights the most popular topics, so people can find them easily.
- Find data for pages that have no views. Find out why people aren’t going there. If the page is important, raise its visibility. If the content is irrelevant, consider removing it entirely. David’s team managed to cut 400 out of 2300 topics.
- Examine the search queries. For example, the number one search query was “transcript”. Discuss this with the product team and ask why people can’t find the transcript. Make the application so user friendly that the docs are not necessary.
Market the data throughout the company. Product managers may be able to glean the features that need more support. Development and QA can see which areas of the product the customers find tricky. Customer support may be able to use your data to support their own findings.
David discussed a few more questions we can ask about the data. He remarked that there’s no such thing as too much data.
Thanks David for a great overview of the type of analytics we can gather and how we can use the data.