Experiences with readers’ comments on the Atlassian documentation wiki

People are what matters in the documentation world. The docs are for the people and increasingly by the people too. One way of giving people a chance to contribute to, and help each other via, the documentation is by allowing them to comment publicly on the doc pages. Susan Grodsky asked me to write something about Atlassian’s experiences with comments on our documentation wiki. Here goes.

There’s a lot to talk about: setting the permissions that determine who can add comments to the pages, managing those comments, the sort of comments people make, how many comments we receive, and above all, is it a good thing or a bad thing to allow public comments on the documentation?

I’ve done some analysis of the comments we received over a week, from 7th to 14th January 2011. This may be a fairly quiet time of year, but it happened to be the period when I had time to do the analysis.

A bit about our documentation wiki

We write and publish all our documentation on a single Confluence site: confluence.atlassian.com, fondly known as CAC. It hosts the documentation for all our products: JIRA, Confluence, Crowd, FishEye, Crucible, Bamboo, JIRA Studio and more.

Each product has its own documentation “space”. Most products in fact have a number of spaces, one for each major release of the product. Many products also have a separate space devoted to developer documentation – APIs, plugin frameworks and the like. If you go to the CAC dashboard and scroll down, you’ll see the list of spaces on the left-hand side of the screen.

And yes, the documentation for the Confluence product is hosted on Confluence itself. The Confluence documentation space is just one of the spaces on CAC.

Stats of visitors to the documentation wiki in a week

To give some context to the number of comments received, I looked at the traffic that the documentation site received in the week of 7th to 14th January 2011.

Across the entire wiki:

• 99 329 people visited the site, in a total of 157 312 visits.
• 45 549 pages were viewed a total of 540 488 times.
• Number of comments on all spaces: 98

Confluence documentation (DOC space) only:

• 1 793 pages were viewed a total of 99 310 times.
• Number of unique page views: 78 037
• Number of comments on this space: 39

My previous blog post has some pretty pictures and more information from Google Analytics.

Analysis of comments on one documentation space in a week

Here are the results of my analysis of the comments received in the week of 7th to 14th January 2011 on the Confluence documentation (DOC space) only.

• Number of comments on the DOC space in that week: 39
• Who made the comments:
  • 18 comments were anonymous – added by someone who was not logged in to the wiki.
  • 21 comments were from logged-in users.
• What the comments were about:
  • Offering information, hints and tips: 9
  • Suggesting an addition/correction to the documentation: 2
  • An unhelpful comment, possibly meant to be humorous: 1
  • Requesting help on functionality (“how do I…” or “is it possible to…”): 9
  • Requesting support (when something does not work as expected): 13
  • Suggesting a feature or improvement in the product: 5

How I tracked the comments

Google Analytics does not show statistics specifically for comments. Instead, I used an RSS feed generated by Confluence to retrieve the comments over the time period I wanted. (To define the RSS feed you want from a Confluence site, click “Feed Builder” on the Confluence dashboard.)

Permissions and public signup

How do we determine who can add a comment? Confluence has a useful permissions scheme. You can grant permissions to specific users, to groups of users, and to “anonymous” users, the last being people who have not logged in.

Each space has its own set of permissions. The permissions determine what people can do with pages, with blog posts, with space administration, and of course with comments. For example, you might choose to allow only a specific group to add and update pages in a given space. That group might be the technical writers, or the company employees. In another space, you might allow everyone to add and update pages, but only a given group to delete pages, and so on.

Talking specifically about comments: For the Atlassian documentation spaces, we have set the permissions to allow everyone to add a comment. Even anonymous users can add comments. Only Atlassian staff members can remove comments.

We have also enabled public signup on the wiki. That means that anyone can sign up for a username and then log in, immediately and at any time. Once logged in, they’re not “anonymous” users any more. They need to enter an email address, but basically they can use any name, username and email address they like.

How we handle the comments

At Atlassian, each technical writer is responsible for a specific set of documentation spaces. We keep a close eye on what’s happening in the documentation, including any comments added to the pages. There are two ways we do that:

• Select the “watch” option on the space. When you are watching a space, Confluence sends you an email notification of all updates in the space, including comments.
• Use an RSS feed, configured to show all activity over a given period of time.
  

As you can see from my analysis above, many of the comments are from people asking for help rather than making suggestions about the documentation itself. The Atlassian Support team monitors the documentation spaces too. The support engineers are really great at responding to comments. We could not do it all without them.

What’s more, readers often help each other. To me, this is one of the most rewarding aspects of allowing public comments.

The technical writers jump in to answer the comments that relate to the documentation. We answer the comment, update the page if necessary and thank the person for their input. A week later we remove the comment and our reply.

Well, that’s the plan anyway 🙂

In practice, we often don’t have time to respond to all the comments appropriately. We also don’t have time to go back a while later and remove all the comments that the support team and other customers have answered.

The result is that many of our documentation pages, especially in the older spaces, have long strings of comments attached.

We’re considering writing a script that we can run regularly to delete comments that are over two years old (or whatever period we choose). The principle is that the page will have been updated a few times and most comments will be irrelevant after such a long period.

If we do start using such a script, we will write up our comments policy so that wiki users know their comments will disappear eventually, even if we haven’t explicitly replied to every comment.

Are people nasty?

We don’t often see malicious comments. There is of course the occasional frustrated person who can’t get something working. We usually ask them to raise a support request or report a bug on the issue tracker, because the documentation is not the place for venting frustration or for getting immediate help with an urgent problem.

What about spam?

Confluence has Captcha for spam prevention. We have configured it to spring into action whenever an anonymous user adds a comment. Captcha prompts the user to read some text from an image and type it into a form. This is very effective in preventing spambots.

We do get the occasional spam comment from real people – often someone trying to sell watches, funnily enough.  They add their advertisements to the pages about wiki notifications and watches. 🙂

We remove spam comments as soon as we spot them.

Some examples of comments

Here are some of the comments we have received. They’re not all from the week of 7th to 14th January 2011. Instead, I had a look around to find some interesting and representative comments all over the wiki. Most are from the DOC space, though the last one is from the CROWD space.

Some people use the comments just to try out the wiki functionality:

Experimenting with wiki comments

Others get a bit more adventurous, and check to see if they can code JavaScript into a comment:

Trying out a bit of JavaScript

People share macros they have written to extend the wiki functionality. This comment was on a page about the {include} macro. Someone has written a related macro:

Sharing a macro

People do tell us off when something goes wrong. This comment was on the page about backing up and restoring a Confluence site:

Telling us off when something goes wrong

Other people jump to our defence. Below is a continuation of the comment above, and a reply from someone else. The reply is the comment that I counted as “unhelpful, possibly meant to be humorous” in my analysis above:

Jumping to our defence

I stepped in to smooth the troubled waters:

Smoothing troubled waters

Here’s a helpful comment about the documentation itself. Someone who shall remain nameless 😉 had forgotten that we’re not in 2010 any more. This comment was on the release notes:

Pointing out a typo in the docs

Next is possibly my favourite string of comments, because it’s so cute and funny and totally off topic. The comments are still on the page:

Been there, done that, got the T-shirt

Readers help each other. To me, that’s the most beneficial aspect of allowing public comments:

Helping each other

A good or a bad thing?

Receiving and responding to comments keeps the documentation alive, dynamic, iterative, constantly improving. The best of agile.

It’s also chaotic, time consuming and open to occasional abuse.

From looking at the comments themselves, I’d say our customers appreciate being able to have their say, ask questions and help each other.

Me? I love it. I would find it hard to go back to a form of technical writing where there’s less contact with the readers. But I do wish I had more time to give due attention to the comments we receive.

Thank you to everyone who comments on and contributes to the wiki documentation. You are stars!