Test-driven API documentation

My latest interest is in innovative and useful ways of doing API documentation. In a comment on a stackoverflow post, Aaron Digulla suggests that we should use tests for the example code in our API documentation.

Here’s the relevant part of Aaron Digulla’s comment:

In my case, I use two tools:
1. A Wiki to give you the big picture…
[snip]
2. Lots of tests which show how you can use the code. You need to write them anyway, so write them just like any other code (clean and readable) and turn them into your examples. Main advantages: Documentation can lie, be wrong or be outdated. Tests Always Tell The Truth (TATTT).

Now, that sounds interesting!

Has anyone else tried this, and do you have any examples that we can take a look at? It would be great to hear how well it works. For example, I wonder if the granularity of the tests matches the granularity required for documentation examples. How do you pick the tests that are relevant for the documentation?

All comments greatly appreciated.πŸ™‚

Going off topic: It’s been a while since we’ve had a tree in one of my posts. Here you go:

Rainblow Lorikeets

Two Rainbow Lorikeets lending colour to a dead tree outside my window


About Sarah Maddox

Technical writer, author and blogger in Sydney

Posted on 17 April 2011, in technical writing, trees and tagged , , , . Bookmark the permalink. 15 Comments.

  1. Interesting additional perspectives from Josh Graham:

    Tweet 1:
    @sarahmaddox for APIs I prefer a specification-level BDD tool like rspec, and for user docs, an acceptance-level BDD tool like Cucumber.

    Tweet 2:
    @sarahmaddox the former gives a dev precise pre-, post-, and invariant conditions for API use. The latter also provides a working demo base..

  2. I think using tests-as-docs works well in trivial cases. A test for a function that computes n elements of the fibonacci sequence is going to be a pretty straightforward example of how to use that function. For example, Python has a kind of test called a doctest, which is a test inside a special kind of comment (a docstring) which describes a function, method, class, module, etc. It’s perfect for these kinds of short, self-contained tests as example code.

    However, as the number of dependencies for a piece of code (like a specific API call) increase (e.g., setup or teardown, database connections, etc.) the more likely it is that the test code is a cartoony example, where major portions are mocked or stubbed out. The test suite is a sort of hall of mirrors where very little code reflects the real world uses. It’s great for finding edge cases and being confident about not (re)introducing bugs, but I wouldn’t use them as examples. If I have to provide a fake example, I’m much more inclined to write one specifically tailored to the documentation, instead forcing in a test case.

  3. As Daniel Beck already mentioned, doctest is one way to start. doctest really embeds small pieces of code.

    One of the advantages of this approach is that it forces you to make the setup for your API really compact. Otherwise, this approach doesn’t really work. If you need 10 lines of setup to get one line of example to work, the 10 lines distract the reader too much.

    At the same time, this means that more complex examples need a different approach. One idea is to upload the test sources along with the documentation on a web server. You must be able to create links to every line in the test sources.

    Now you can add special comments to your test sources to pick a block of code. That way, you can read them from the actual test code when you generate the HTML documentation. This allows you to demonstrate more complex API (which needs more setup) in your docs without distracting the reader. If they want the complete working example, they can follow the link.

    At the same time, this makes sure that the example really works.

    The third approach is to mix source and documentation. Donald Knuth has used this with great success while writing TeX. This is called “literate programming“.

    The main problem with literate programming (LP) is that it hard to separate concerns. When you write documentation, you can have several hats: Either you write it for someone who has to maintain the software or for someone who wants to use it. As a user, you don’t need to know too many details about the inner working. With LP, that’s hard to hide since the software is actually compiled from the documentation.

    OTOH, LP forces you to approach many problems from a different angle (like Test-driven development): Before you write the code, you look at it from a more abstract level. That can help to clear your mind.

    All in all, I think our main problem is that computers aren’t powerful enough. Right now, we separate source and docs. What we want it a simple way to attach additional information to any part of the source code. Without cluttering the view. There must be ways to view at the code from different angles. For example, a developer will want to see the pure source while someone from the documentation team will want to know when a developer changes something in the source that will break the docs. The tech writers want to be able to mark a piece of code as “used in the docs” or be able to open the project with a “documenters view” which hides most of the source coded but shows all the embedded documentation.

    But since we still use plain ASCII editors, it’s hard to add/hide information bits everywhere.

  4. In another life, I did use test code as snippets IF the test was designed to work and not designed to find flaws in the SDK or API.

  5. I like the idea of a language expecting tests as doctest does with Python. I’ve also tried adding examples to javadocs for a Java API and it was painful. Something that would be very helpful would be for the API documentation to link to the source code of tests that use that method or class. Perhaps “Links to three uses of this method”. Lots of tests use mock classes though which may not be as helpful as seeing how the method is used in the product.

  6. Hallo Daniel, Aaron, Bill and Matt

    Thanks for all the ideas and feedback so far! Lots of good reading to do.

    On the subject of writing “hibrid” documentation, where part is written by a person and part is drawn from a source repository:

    We use Confluence wiki for our documentation. There’s a useful “snippet” macro for Confluence, supplied by the Snippet plugin.

    Here’s an extract from the plugin description:
    A Confluence plugin to retrieve and display a snippet of text from a remote URL (e.g., Subversion). The content will be trimmed down to the content found between some special tags. This is very handy for documentation that includes code. The code can be in Subversion (whatever build system is used will make sure it builds properly) and the URL can be a Subversion URL.

    Here’s an example of a tutorial that uses code snippets drawn from the source:
    http://confluence.atlassian.com/display/DEVNET/Plugin+Tutorial+-+Writing+REST+Services

    As you can see, it works pretty well. The problem we still need to solve, is version awareness. Over time, the code in the source repository will move to a new version, with the result that it no longer matches the words on the documentation page.

    • That’s pretty simple to solve:

      Write tests which specifically test whether the code behaves as described in the docs. Let’s call them “doc unit tests”. Use those tests with the Snippet plugin in the docs – that should work well since those tests demonstrate exactly what you want to explain in the docs.

      Now you can create another confluence plugin that collects all places where the Snippet plugin is used. Write the result to a file or a new wiki page.

      When a test fails and you need to change it, this wiki page shows you which docs to update.

    • For an example of “doc junit tests”, see my XML parser: http://code.google.com/p/decentxml/wiki/Tutorial

      I’m just missing a Snippet plugin in Google Code, so it’s manual copy and paste.

      • Hallo digulla

        Thanks for the link. Nice tutorial. I love the chatty style and the abundance of code examples.

        I’m not sure whether you prefer to be called Aaron or digulla, so I’m going with the signature on the comments. I hope that’s OK.πŸ™‚

        Cheers, Sarah

  7. Sarah,

    Nice snippet example. Does that feature also work if you go all meta and try to reference some file with wiki markup in it?

    ~Matt

    • Hallo MattπŸ™‚
      Hah, I don’t know. Knowing the quality of Don’s work, I’d guess it would work. Care to try it out and let us know what happens?πŸ˜‰
      Seeya, Sarah

  8. Hi, great discussion!
    For Delphi and C++Builder we’ve defined 3 types of code info: snippet, example, sample. Snippets are most like unit tests; very simple. Examples pull snippets into a commonly-used context such as a form. Samples are full-on, multi-page apps. We provide compilable examples and samples on SourceForge so customers can reuse them. Samples are the foundations of our step-by-step tutorials.

    Writers can reuse unit tests for snippets. On the other hand, QA can use our examples and samples for more robust testing!

    -Dee

    • Hallo Dee

      That sounds very good indeed. I’ve had a look around your documentation wiki. Wow, it’s very comprehensive and very well laid out!

      I found a number of tutorials about how to use the IDE, but I couldn’t find API-type tutorials. Would you be able to post a link to a page that uses the snippets, examples and samples?

      Cheers, Sarah

  1. Pingback: Search tools for API documentation « ffeathers — a technical writer’s blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: