Blog Archives

Should we allow comments on documentation pages

This is a very interesting question: Should we, as technical writers, allow comments on our documentation pages? It’s interesting because it’s a multi-faceted question, and because people have such strong feelings about it. My quick answer is, “Yes”. Ha ha, but there’s always a “but” or two. Read on, and then I’d love to know what you think.

I’m a technical writer at a company called Atlassian. We write all our product documentation on a wiki. For example, here is the Confluence user’s guide. What’s more, we have configured the wiki permissions to allow anyone to add comments to the pages. (We also allow known contributors to update the pages – but that’s another story.) Every now and then, the question comes up for debate again: Is it a good thing to allow comments on the pages? Up to now, we have decided each time to keep the comments. (Well, except for the developer documentation. More below.)

Multi-faceted question

We could rephrase the question like this:

Should we ask for feedback on the documentation? If so, are comments the best way of getting that feedback?

Or like this:

Should we allow everyone to comment on the documentation, or just known people?

Or:

What benefits do our customers get from being able to add and read comments on the documentation?

Or:

What benefits do we, as technical writers, get from the comments people add? And do we suffer any pain?

Mmmm…

What  benefits does the organization get from allowing people to add comments to the documentation?

And so on.

Examples of comments

On the Atlassian product documentation, we allow everyone to add comments. Even people who have not logged in to the wiki – their comments  are recorded as “anonymous”.

Some comments are very relevant to the documentation itself. For example, the comments on this page about supported platforms in the Confluence administrator’s guide:

Some of the comments on the supported platforms page

Some of the comments on the supported platforms page

Other comments can be requests for help, or suggestions for new features or improvements in the product. Often a reader will add some information that will be useful to other readers. This one is on the page about configuring Tomcat’s URI encoding:

Page about configuring Tomcat's URI encoding

A reader offers information to others

Advantages of allowing comments

A quick list:

  • People tell us about errors or gaps in the documentation.
  • People use the documentation as a tool to help each other.
  • We learn about new ways that people are using our products, and sometimes even about new ways that they want to use the documentation.
  • The documentation becomes an active hub of interesting information. Readers receive notifications when other people add comments, and so keep coming back to the documentation to see the latest. The documentation sticks in their minds as a good place to find what they need.

Disadvantages of allowing comments

Another quick list:

  • Comments can make a page hard to read, if there are too many of them.
  • People may add worthless comments, spam, or even nuisance comments.
  • It is time-consuming, if not impossible, to respond to all the comments.
  • It is time-consuming, if not impossible, to clean up the comments when they are no longer needed. (But we could automate this.)

Please add a comment 😉 to this blog post, telling me what I’ve missed in these two lists.

No more comments on the developer documentation

A few months ago, we moved all of the developer documentation to a different wiki, the Atlassian Developers site. By “developer documentation”, I mean the API reference guides, the plugin development tutorials, and all the stuff related to developing add-ons and extensions for our products.

The developer documentation is still on a wiki. In fact, it’s a Confluence wiki just like the product documentation. But we have customised the look and feel, and added some plugins. The thing to note, from the point of view of this post, is that we have turned off the page comments. Instead, there are two panels at the bottom of each page:

Answers and feedback panels on documentation page

Atlassian Answers and feedback panels

The “Atlassian Answers” panel shows a list of posts drawn from a discussion forum called Atlassian Answers. The panel is supplied by a plugin for the wiki that hosts the documentation. The plugin matches the labels on the documentation pages to the tags on the discussion forum. The matching process is not perfect. In particular, labels and tags are not the most reliable way of matching content from a discussion forum and a documentation site. We plan to improve this, by a smarter matching of page titles and content.

The “feedback” panel is supplied by another plugin. When a person clicks “Yes”, “Somewhat” or “No”, a dialog appears where they can give us more information. The plugin posts an email containing that feedback, which in turn triggers the creation of an issue on our JIRA issue tracker. The technical writers and developer relations team can then assess and react to the feedback.

Comparing comments and feedback forms

An advantage of comments over feedback forms is that readers can see and respond to each other’s comments. People can benefit from the advice of other readers. They can hold a conversation and help each other solve problems, quite independently of the documentation authors.

Comparing comments and discussion forums

An advantage that comments have over discussion forums is that the comments are right there on the relevant page. People do not need to go looking for them on a separate discussion site. The information in the comments complements what is on the page.

On the other hand, a discussion forum is in itself a repository of information. I guess, as a technical writer, I’d like to keep people’s attention on the documentation. I’d prefer it if the docs did not become a dead site, for reference only.

But what serves the customer better?

What do you think?

Heh heh, it’s a complex question. And now let me add this specific question to the list: Would you cry if we removed the ability to comment on the Atlassian documentation in particular? (This is just a general question. There are no definite plans at the moment to remove commenting.)

Let the comments begin. 🙂

External tools for getting feedback on your documentation

A tweet from Pamela Fox tweaked my curiosity. She mentioned KISSinsights as a good tool for getting readers’ feedback on documentation. It looks pretty neat. I’m wondering what other, similar tools are available and whether technical writers have much experience with them.

A number of platforms offer inbuilt tools for getting readers’ feedback. Confluence offers comments and various feedback and rating plugins. Mindtouch and RoboHelp have their own feedback mechanisms, as do other platforms.

But I’m interested in external tools where you can craft a feedback form and then embed it into your documentation. KISSinsights is the tool mentioned by Pamela Fox. Once installed on your site, KISSinsights pops up a panel  from the bottom of your page, asking the reader some questions. If you pay for a  licence you can customise the questions, otherwise you get the questions that come with a standard KISSinsights installation. The KISSinsights FAQ page is pretty good.

Another example is Wufoo, recently acquired by SurveyMonkey. You can build a form on Wufoo and then grab the HTML to embed into your page. When readers fill in the form, the information is stored in the Wufoo database and you can extract the results from there.

Does anyone have good or bad experiences with these and other tools?

Experiences with readers’ comments on the Atlassian documentation wiki

People are what matters in the documentation world. The docs are for the people and increasingly by the people too. One way of giving people a chance to contribute to, and help each other via, the documentation is by allowing them to comment publicly on the doc pages. Susan Grodsky asked me to write something about Atlassian’s experiences with comments on our documentation wiki. Here goes.

There’s a lot to talk about: setting the permissions that determine who can add comments to the pages, managing those comments, the sort of comments people make, how many comments we receive, and above all, is it a good thing or a bad thing to allow public comments on the documentation?

I’ve done some analysis of the comments we received over a week, from 7th to 14th January 2011. This may be a fairly quiet time of year, but it happened to be the period when I had time to do the analysis.

A bit about our documentation wiki

We write and publish all our documentation on a single Confluence site: confluence.atlassian.com, fondly known as CAC. It hosts the documentation for all our products: JIRA, Confluence, Crowd, FishEye, Crucible, Bamboo, JIRA Studio and more.

Each product has its own documentation “space”. Most products in fact have a number of spaces, one for each major release of the product. Many products also have a separate space devoted to developer documentation – APIs, plugin frameworks and the like. If you go to the CAC dashboard and scroll down, you’ll see the list of spaces on the left-hand side of the screen.

And yes, the documentation for the Confluence product is hosted on Confluence itself. The Confluence documentation space is just one of the spaces on CAC.

Stats of visitors to the documentation wiki in a week

To give some context to the number of comments received, I looked at the traffic that the documentation site received in the week of 7th to 14th January 2011.

Across the entire wiki:

  • 99 329 people visited the site, in a total of 157 312 visits.
  • 45 549 pages were viewed a total of 540 488 times.
  • Number of comments on all spaces: 98

Confluence documentation (DOC space) only:

  • 1 793 pages were viewed a total of 99 310 times.
  • Number of unique page views: 78 037
  • Number of comments on this space: 39

My previous blog post has some pretty pictures and more information from Google Analytics.

Analysis of comments on one documentation space in a week

Here are the results of my analysis of the comments received in the week of 7th to 14th January 2011 on the Confluence documentation (DOC space) only.

  • Number of comments on the DOC space in that week: 39
  • Who made the comments:
    • 18 comments were anonymous – added by someone who was not logged in to the wiki.
    • 21 comments were from logged-in users.
  • What the comments were about:
    • Offering information, hints and tips: 9
    • Suggesting an addition/correction to the documentation: 2
    • An unhelpful comment, possibly meant to be humorous: 1
    • Requesting help on functionality (“how do I…” or “is it possible to…”): 9
    • Requesting support (when something does not work as expected): 13
    • Suggesting a feature or improvement in the product: 5

How I tracked the comments

Google Analytics does not show statistics specifically for comments. Instead, I used an RSS feed generated by Confluence to retrieve the comments over the time period I wanted. (To define the RSS feed you want from a Confluence site, click “Feed Builder” on the Confluence dashboard.)

Permissions and public signup

How do we determine who can add a comment? Confluence has a useful permissions scheme. You can grant permissions to specific users, to groups of users, and to “anonymous” users, the last being people who have not logged in.

Each space has its own set of permissions. The permissions determine what people can do with pages, with blog posts, with space administration, and of course with comments. For example, you might choose to allow only a specific group to add and update pages in a given space. That group might be the technical writers, or the company employees. In another space, you might allow everyone to add and update pages, but only a given group to delete pages, and so on.

Talking specifically about comments: For the Atlassian documentation spaces, we have set the permissions to allow everyone to add a comment. Even anonymous users can add comments. Only Atlassian staff members can remove comments.

We have also enabled public signup on the wiki. That means that anyone can sign up for a username and then log in, immediately and at any time. Once logged in, they’re not “anonymous” users any more. They need to enter an email address, but basically they can use any name, username and email address they like.

How we handle the comments

At Atlassian, each technical writer is responsible for a specific set of documentation spaces. We keep a close eye on what’s happening in the documentation, including any comments added to the pages. There are two ways we do that:

  • Select the “watch” option on the space. When you are watching a space, Confluence sends you an email notification of all updates in the space, including comments.
  • Use an RSS feed, configured to show all activity over a given period of time.

As you can see from my analysis above, many of the comments are from people asking for help rather than making suggestions about the documentation itself. The Atlassian Support team monitors the documentation spaces too. The support engineers are really great at responding to comments. We could not do it all without them.

What’s more, readers often help each other. To me, this is one of the most rewarding aspects of allowing public comments.

The technical writers jump in to answer the comments that relate to the documentation. We answer the comment, update the page if necessary and thank the person for their input. A week later we remove the comment and our reply.

Well, that’s the plan anyway 🙂

In practice, we often don’t have time to respond to all the comments appropriately. We also don’t have time to go back a while later and remove all the comments that the support team and other customers have answered.

The result is that many of our documentation pages, especially in the older spaces, have long strings of comments attached.

We’re considering writing a script that we can run regularly to delete comments that are over two years old (or whatever period we choose). The principle is that the page will have been updated a few times and most comments will be irrelevant after such a long period.

If we do start using such a script, we will write up our comments policy so that wiki users know their comments will disappear eventually, even if we haven’t explicitly replied to every comment.

Are people nasty?

We don’t often see malicious comments. There is of course the occasional frustrated person who can’t get something working. We usually ask them to raise a support request or report a bug on the issue tracker, because the documentation is not the place for venting frustration or for getting immediate help with an urgent problem.

What about spam?

Confluence has Captcha for spam prevention. We have configured it to spring into action whenever an anonymous user adds a comment. Captcha prompts the user to read some text from an image and type it into a form. This is very effective in preventing spambots.

We do get the occasional spam comment from real people – often someone trying to sell watches, funnily enough.  They add their advertisements to the pages about wiki notifications and watches. 🙂

We remove spam comments as soon as we spot them.

Some examples of comments

Here are some of the comments we have received. They’re not all from the week of 7th to 14th January 2011. Instead, I had a look around to find some interesting and representative comments all over the wiki. Most are from the DOC space, though the last one is from the CROWD space.

Some people use the comments just to try out the wiki functionality:

Experiences with readers’ comments on the Atlassian documentation wiki

Experimenting with wiki comments

Others get a bit more adventurous, and check to see if they can code JavaScript into a comment:

Experiences with readers’ comments on the Atlassian documentation wiki

Trying out a bit of JavaScript

People share macros they have written to extend the wiki functionality. This comment was on a page about the {include} macro. Someone has written a related macro:

Experiences with readers’ comments on the Atlassian documentation wiki

Sharing a macro

People do tell us off when something goes wrong. This comment was on the page about backing up and restoring a Confluence site:

Experiences with readers’ comments on the Atlassian documentation wiki

Telling us off when something goes wrong

Other people jump to our defence. Below is a continuation of the comment above, and a reply from someone else. The reply is the comment that I counted as “unhelpful, possibly meant to be humorous” in my analysis above:

Experiences with readers’ comments on the Atlassian documentation wiki

Jumping to our defence

I stepped in to smooth the troubled waters:

Experiences with readers’ comments on the Atlassian documentation wiki

Smoothing troubled waters

Here’s a helpful comment about the documentation itself. Someone who shall remain nameless 😉 had forgotten that we’re not in 2010 any more. This comment was on the release notes:

Experiences with readers’ comments on the Atlassian documentation wiki

Pointing out a typo in the docs

Next is possibly my favourite string of comments, because it’s so cute and funny and totally off topic. The comments are still on the page:

Experiences with readers’ comments on the Atlassian documentation wiki

Been there, done that, got the T-shirt

Readers help each other. To me, that’s the most beneficial aspect of allowing public comments:

Experiences with readers’ comments on the Atlassian documentation wiki

Helping each other

A good or a bad thing?

Receiving and responding to comments keeps the documentation alive, dynamic, iterative, constantly improving. The best of agile.

It’s also chaotic, time consuming and open to occasional abuse.

From looking at the comments themselves, I’d say our customers appreciate being able to have their say, ask questions and help each other.

Me? I love it. I would find it hard to go back to a form of technical writing where there’s less contact with the readers. But I do wish I had more time to give due attention to the comments we receive.

Thank you to everyone who comments on and contributes to the wiki documentation. You are stars!

%d bloggers like this: