Author Archives: Sarah Maddox

Analytics strategies for evaluating and planning doc updates

Over the past few months, I’ve been delving into analytics and feedback on the doc site that I currently manage. I’m crafting strategies as I go, and creating reports for product stakeholders to get their input too. I hope some of the strategies described in this post may be useful or at least interesting to other people who’re looking into how to use analytics.

Note: Although I work at Google, this post does not constitute any recommendations on the use of any Google product. I’m a technical writer, and I’m using analytics and feedback in the same way other tech writers do, to gain insights into the doc set that I manage. I am by no means an expert on analytics.

Let’s get some technical details out of the way first. The doc site under discussion is kubeflow.org, which hosts the documentation for an open source machine learning platform called Kubeflow. The documentation is also open source. The source for the docs lives on GitHub.

I’m using Google Analytics to see the doc usage stats. The Kubeflow doc site is fairly new. I enabled Google Analytics and the feedback widget on February 27, 2019, which means that the stats start from that date.

To gather user ratings on the doc pages, I’m using the feedback widget that’s available with the Docsy theme. The Kubeflow website uses Docsy and Hugo. If you’re interested in the details of the website tooling, take a look at the website README.

Goals for the analytics reports

The Kubeflow community and I are interested to see how people are using the docs. A high percentage of page views in a particular area can indicate a high level of interest in the related product features, or can point to an area of the product where people need more help than in other areas.

From a docs point of view, my goal is to identify the top priority docs for improvement, and to get some direction on the types of improvements we need to make. For example, if people are particularly interested in an area of the docs, and at the same time are not satisfied with the information they find there, then that area of the docs is high priority for improvement.

Overall site views

I started by looking at the number of website views from March (when Google Analytics became available on the site) to November (now). The number of views per month has more than doubled in that time, from 104,000+ to 220,000+. It’s good to know our reader base is increasing.

Total website views

Most-viewed pages

I looked at the pages with highest number of views across the site as a whole, and also within a few high-priority sections of the docs.

The period for these stats is two months, from September 1 – November 1. Our previous report was in July. I didn’t include August in the stats, because we did some information architecture refactoring in August. We moved many pages around. Moving pages affects the Google Analytics stats, which makes August a bad month to use for assessments in this case.

Most-viewed pages

The top entry, “/”, denotes the Kubeflow website home page: https://www.kubeflow.org/. This page consistently receives the highest number of views.

As in previous reports, the second-most viewed page is the main Getting Started guide. It’s linked from the website home page. Other getting-started guides rank highly too.

Also as in previous reports, the third-most viewed page is About Kubeflow. It’s linked from the top-level menu bar with text “What is Kubeflow”.

In a change from previous reports, the Use Cases section has replaced components and notebooks in the list of 10 most-viewed sections. I should start paying attention to this section.

Other pages in the top 10 are the same as in previous reports: the docs index page and the pipelines section.

Strategies for most-viewed sections and pages

My overall strategy for the top-viewed pages is to spend time perfecting the user experience on those pages, addressing any issues, and making sure people find the information they need:

  • Improve the textual and visual content of the most-viewed pages. For example, we recently ran a doc sprint in which we spent considerable time restructuring and rewriting the website home page, which is the most highly viewed page in the doc set. Feedback on the new design and content is good.
  • Link from the most-viewed pages to content deeper in the site, to ensure people find all the information they need. For example, we recently rewrote the “About Kubeflow” page and added links down into relevant content on the site.
  • Examine the bounce rate and time on page, to see how people are using the page.
  • Examine feedback, to see whether people are finding the content useful.

Getting feedback from readers

Every page on the Kubeflow website has a feedback option. The option asks “Was this page helpful? Yes / No”.

  • About Kubeflow received the most feedback, and 24 of 28 responses (85.7%) were positive. That’s an improvement from the July analysis, which showed 70% positive.
  • Getting Started received the second-most feedback, and 11 of 15 responses (73%) were positive. That’s exactly the same as in July.

It’s worth noting that the number of feedback responses is very low in comparison with the number of page views. Also, people are more likely to respond with negative feedback than with positive. Even so, the feedback is useful, particularly when it’s strongly positive or negative, and if the ratios of positive to negative change after we’ve updated the content.

Deep dive into specific sections and pages

Based on the above statistics and feedback results, I examined some specific pages in greater detail.

The next screenshot shows the 10 most-viewed pages within the getting-started section. We reorganized this section significantly in August. It’s useful to see which getting-started experiences are the most often viewed, in the period since that significant refactoring.

The guide to deploying Kubeflow on an existing Kubernetes cluster (roughly equivalent to on-premises installation) has most views. The workstation installation guides come next, followed by deployment to a cloud.

The following stats are for the Getting Started page, which introduces the getting-started section:

Looking at the information for this getting-started overview page in detail:

  • The page has the second-highest number of page views in the entire doc set (the top-level kubeflow.org page has highest).
  • Bounce rate* has continued dropping, from 56% in April to 44.15% in July, to 39.6 percent now. That’s a great improvement. Our goal was see it drop below 40% – goal achieved!
  • Time on page is 1 minute 7 seconds. That’s fine. There’s no need for people to spend longer on the page, because this is an overview page and the meaty content is in sub-pages.
  • The getting-started overview page has received the second-highest amount of feedback of all pages on the site , and 11 of 15 responses (73.3%) were positive. That’s exactly the same as in July.
  • Overall, the getting-started pages continue to receive low ratings.

* The bounce rate for a page is the percentage of user sessions that started and ended with that page. So, people entered the site on that page, and left without viewing any other pages. I’ve seen guidelines indicating that, as a general rule, we should avoid a bounce rate higher than 70%. If many people visit a page but leave immediately, this may indicate that the page isn’t giving them what they need, and so they leave the site. (It does depends on the type of page. The purpose of some pages is exactly to send people elsewhere.)

We need to improve the content of the getting-started section so that it better meets the readers’ expectations. One tactic I hope to follow, if I can get time from a UX research team, is to test the pages with some specific groups of users. In addition, I’ve already seen feedback from customer issues that people are looking for a single, recommended flow for getting started quickly. Currently the docs offer all the options, but don’t give much guidance on where to start.

Next up is the About Kubeflow page:

Looking at the About Kubeflow page in detail:

  • It’s the third-most highly viewed page on the website.
  • In previous reports, bounce rate came down from 63% in April to 60.4% in July. Bounce rate has now gone up again to 62%. We need to lower the bounce rate, as this page is a highly-viewed page and we want to draw people deeper into the site. I’m working on a new Kubeflow overview (pull request #1339). When that new page is available, I’ll link to it from the About Kubeflow page, and then re-assess the bounce rate.
  • Average time on page is two minutes. That’s good for an overview page. People are engaged in the content.
  • The page has received the most feedback of all pages on the site, and 24 of 28 responses (85.7%) were positive. That’s an improvement on July (70% positive).
    We refactored the page in June to provide more information and links. I hope to improve the positivity still further by linking to the new Kubeflow overview mentioned above.

What about the Use Cases section, which has recently made it into the top 10 most highly viewed sections?

  • It’s interesting to see a set of guides arrive in the top 10 most highly-viewed pages for the first time. This change potentially indicates that our audience is maturing and looking for more in-depth use-case focused docs. The product (Kubeflow) is relatively new, and is currently working towards a v1.0 launch in 2020. Up now, perhaps most people have been focused on getting the product up and running and trying the simple use cases provided in the getting-started section. Now maybe they need more in-depth use cases.
  • The feedback ratings on this section are low. We need to make sure people get what they’re looking for.
  • One action I’m considering is to adjust the information architecture to reflect what people are probably looking for. At least in the short term, I could rename the section, as it describes highly specific ways of using the product, rather than the more generic use case information that people may be looking for. Alternatively, I could move the content into another section, such as the “further setup and trouble shooting” section.
  • Then, when we have more bandwidth and have had time to do more research, we should flesh out the section with more use cases. We do already have some good examples and tutorials, which we can include in this section.

Open source contributors to the docs

Moving from Google Analytics to GitHub stats for the doc repository, it’s interesting to see the fluctuation in the number of contributors to the docs. It’s not just me writing the docs!

The following events influenced the contributor numbers:

  • We ran a community-wide Kubeflow doc sprint in July. Contributions increased significantly during that period, and stayed high for a while afterwards.
  • Contributions picked up towards the Kubeflow v0.7 release, which happened in early November.
  • In mid November, we ran a doc fixit for external tech writers at the Write the Docs conference in Australia. That fixit causes the large spike at the right-hand edge of the graph.

We need to run more doc sprints and fixits!

Traffic sources

A product stakeholder requested information about the sources of website traffic. I haven’t yet figured out any related strategies.

In the period August 1 – November 1, 2019, close to 60% of the website traffic came from organic search. Referrals accounted for 22%.

Traffic sources

I looked at the referrals, and found that the largest percentage (29%) of referrals come from GitHub. This is not surprising, given that the source code for the product is also on GitHub. The next-largest percentage is 8.5%, coming from a related doc site.

The top 10 search terms are primarily variations of the product name, “Kubeflow”, with one outlier: “minikf” at number 4. MiniKF is a deployment tool for Kubeflow.

Search terms

More analytics tips?

If you have any analytics tips or experiences to share, I’d love to hear them. Links are welcome!

From the Tech Writing 101 workshop at Write the Docs AU 2019

Sydney hosted the annual conference of Write the Docs Australia this week. As part of the conference, I ran a Tech Writing 101 workshop. It was a very rewarding experience. If you ever consider running a conference or workshop for a group of technical writers, do it! Tech writers are an engaged, humour-loving, smart group of people.

The workshop

The workshop teaches the principles and patterns of effective technical writing. Before the event, the participants do some pre-reading. Then, during the two-hour workshop, we do a series of exercises and discussions based on the principles in the pre-reading. This is a good way of cementing the patterns into your brain. The next time you write an overlong sentence, or use the passive voice, you’re likely to recognise the anti-pattern and do something to correct it.

We had around 45 participants at the workshop during the conference. Here’s a shot of the room during the workshop. At this stage, the participants had just finished one of the exercises and were discussing their solutions with their partners:

Three assistants helped with running the workshop. They walked around the room answering questions, assisting with logistics, and generally making sure everyone had a good experience. A big thank you to:

The Tech Writing 101 workshop was developed by tech writers at Google to train engineers and others in the principles of effective technical writing. Google is currently preparing a revised, improved set of pre-reading and presentation content, which will be available for tech writers all over the world who want to run the workshop. Stay tuned for news on this front.

What else happened at the conference?

Write the Docs Australia 2019 was jam-packed with talks, workshops, lightning talks, and unconferences. Take a look at the full program.

Here’s the Twitter hashtag: #wtdau2019.

Thanks so much to all the organisers and attendees. Write the Docs AU 2019 was awesome. See you at Write the Docs AU 2020!

Results of open source doc fixit at Write the Docs Australia 2019

This week saw the 2019 conference of Write the Docs Australia. I was delighted to be able to take part in this event. So many good discussions, interesting talks, and lovely people!

On the Thursday afternoon of the conference, I ran an open source doc fixit. A doc fixit is an event where people get together to fix problems in a set of documentation. In the space of just two hours, a group of Write the Docs attendees contributed polish and shine to the Kubeflow documentation. Two fellow tech writers from Google assisted at the event: Alec Glassford  and Riona MacNamara.

Fixit results

Together the fixit participants fixed 33 bugs and merged 31 pull requests (PRs). Thirty-three sets of textual and formatting improvements. That’s a really great contribution to the Kubeflow open source doc set.

The screenshot shows some of the PRs submitted on GitHub by the fixit participants. A pull request (PR) is a set of changes that a contributor is presenting for merging into the open source project:

What do the resulting docs look like? One of the pages that received tech writer love was the guide to authenticating Kubeflow to GCP:

The contributors

Approximately 20-25 people took part in the fixit. A few people paired up to work together, which resulted in 15 individuals submitting pull requests (PRs). Some people submitted two or three PRs, leading to a total of 31 PRs.

The participants were a diverse group. Most were technical writers, given that the fixit was part of a Write the Docs conference. A few people had prior experience of GitHub, but most were new to open source and GitHub.

Goals and tactics

My goal for this fixit was to ensure that the participants had an enjoyable experience as well as the opportunity to learn a bit about open source processes and GitHub technology. I knew that the participants would have no knowledge of the product that the docs cover (Kubeflow).

Another important goal was to contribute meaningful improvements to the Kubeflow docs. Kubeflow is working towards a v1.0 release in January 2020. In the open source world, v1.0 is an important milestone, bringing certain guarantees of stability and polish. The v1.0 docs need to be polished too.

With the above in mind, I decided on a few tactics:

  • Keep the fixes relatively simple: typos, list formatting, broken links, outdated content. I did sprinkle in a couple of more complex fixes, where people needed to trawl through other PRs or issues to gather information. Those were for the folks who were already comfortable on GitHub.
  • Describe the fixes in detail, so that people didn’t inadvertently change the meaning of a sentence when polishing the syntax.
  • Make sure that only one person would submit a PR for a given page. I wanted to avoid messy merge conflicts, which may be frustrating and may even be difficult to resolve in the course of a two-hour fixit. I created a spreadsheet with one row per page, listing all the fixes required on that page. The fixit participants would then put their names next to one or more rows, to claim the pages they would work on.
  • Have three of us (Riona, Alec, and myself) walking around the room and assisting with the tricky bits.
  • Have the three of us reviewing and approving PRs as they came in, so that the participants could see the results of their work published in the doc set as soon as possible.
  • Provide a detailed guide for participants, including the basics of how to work in GitHub and how to update the Kubeflow docs. I wrote the guide, then asked Alec to test it. It was a good thing that he did test the guide, because it turns out that the GitHub experience is quite different for someone who doesn’t have contributor rights to a repository (Alec didn’t yet have them when he tested the guide) as opposed to someone who does (I’m an administrator of the GitHub repo and a member of the Kubeflow organisation).
  • Direct participants to use the GitHub web interface rather than the git command line interface.
  • Attempt to review and approve the PRs during the course of the fixit, so that participants could see the results of their work in the repository and on the docs website. To make this happen, I granted Alec and Riona the necessary review and approval permissions on the GitHub repository where the docs reside. We’ll remove those permissions now that the fixit is over.

What happened at the fixit

At the start of the fixit, I presented a short talk introducing the concepts of open source software and documentation, and the goals of the Kubeflow project.

Then the participants chose a page to work on, based on the spreadsheet we had prepared, and started work.

  • Their first task was to create an issue in the GitHub repo where the docs reside, to track their work.
  • Their second task was to find the page that needed fixing, and open the page in edit mode.
  • After making the necessary updates, the participants submitted a PR for review.

Alec, Riona, and I moved around the room, answering questions and helping people do the initial signups and then submit their issues and PRs.

In between answering questions, we also reviewed the incoming PRs and approved each one when it was ready for merging into the docs repository on GitHub. The continuous merge/publish tools on the GitHub project merged the change and published the update in the docs.

Hiccups and learning

All in all, things went smoothly. I’m very happy with the results. Huge thanks to the participants and to Alec and Riona for their help both before and during the fixit.

Alec, Riona and I were pretty busy helping the participants with various signup processes, GitHub idiosyncrasies, and other tricky bits. We didn’t have much time for reviewing and approving PRs. Next time, more assistants!

A few participants were confused about the difference between an issue and a PR in the GitHub repository. This reminds me how easy it is to succumb to the curse of knowledge: the fixit guide explained why it’s a good idea to create an issue (thanks Alec for suggesting this addition!) but it didn’t go into detail about the difference between an issue and a PR. If both those things are new to you, the difference is not obvious, particularly since they look very similar on GitHub.

A number of participants commented that the initial process of starting to contribute was lengthy and a bit bumpy. People need to sign up to GitHub, sign the contributor licence agreement (CLA), open a page for editing, then follow the GitHub prompts to submit the change, then create a PR (twice, seemingly, as the UI is a little clumsy). People were delighted to know that much of the process is once-off, and their second PR went more smoothly.

On a side note: We were using the GitHub UI because it’s initially a simpler experience than the git command line. Personally, I much prefer the command line. It’s restful. When I sit down with my laptop and type git status, it’s as if I’m saying:

Hey git, it’s been a while. I got distracted by messier things, but I know you’ve been keeping track of the important stuff for me. Let’s get going.

Feedback from participants

Fixit participants said that they appreciated the opportunity to learn about open source and to have their first experience of working in GitHub.

During the general feedback at the close of the conference, people mentioned that they were pleased some parts of the conference involved active participation, instead of listening passively to talks for the full two days.

What else happened at the conference?

Write the Docs Australia 2019 was jam-packed with talks, workshops, lightning talks, and unconferences. Take a look at the full program.

Here’s the Twitter hashtag: #wtdau2019.

Thanks so much to all the organisers and attendees. Write the Docs AU 2019 was awesome. See you at Write the Docs AU 2020!

Join the Kubeflow doc fixit at Write the Docs AU conference

Are you coming to the Write the Docs Australia 2019 conference on 14-15 November in Sydney? You’re invited to join us in a two-hour doc fixit on Thursday afternoon, as part of the conference.

Become a contributor to an open source project, learn a bit about how open source works, and help improve the experience of Kubeflow users. All in just two hours!

During the fixit, you’ll add a touch of tech writer shine to the Kubeflow docs. Docs are a super important part of the user experience of any product. Typos and grammatical inconsistencies can spoil that UX. Yet typos and odd syntax creep into any doc set so easily, especially when the doc set is largely written by non tech writers. You can help us set things right.

Where and when

The doc fixit is part of the Write the Docs Australia 2019 conference.

Registration

You don’t need to register separately for the doc fixit. Just register for the conference, then come along to the fixit on Thursday.

Your friendly doc fixit helpers

The doc fixit hosts are:

What happens at the fixit

Here’s how the fixit will work:

  • Before the fixit, I’ll create a spreadsheet with a list of doc bugs that need fixing. They’ll mostly be small things: typos, consistency in page structure, capitalisation, and so on.
  • At the start of the fixit, I’ll give a very short talk introducing the product (Kubeflow) and open source.
  • Then the group will look at the list of bugs and each person will choose what they want to do.
  • My assistants and I will help people create GitHub IDs if necessary.
  • Each person will create an issue in the GitHub issue tracker, describing the bug they’re about to fix.
  • Each person will then update the relevant doc on GitHub and send a pull request (PR) for review.
  • My assistants and I will help people sign the contributor licence agreement if necessary. (A bot will prompt them to do this when they send their first PR.)
  • My assistants and I will review the pull requests and approve each one when it’s ready.
  • The continuous merge/publish tools on the GitHub project will merge the change and publish the update in the docs.
  • The contributor will see their update appear in the docs!

I’ll also prepare a guide to for fixit participants, with the basics on how to work in GitHub and how to update the Kubeflow docs. The guides, in combination with the three of us helping during the fixit, should make the fixit fun and a useful learning experience for everyone.

Prerequisites

Here’s how you can prepare for the Kubeflow doc fixit:

  • Bring a laptop with WiFi capabilities.
  • If you don’t already have a GitHub account, sign up for one. If you have time to do this before the start of the sprint, that’s great. If not, you can do it during the sprint.
  • Sign the Google Contributor License Agreement (CLA). If you have time to do this before the start of the sprint, that’s great. If not, you can do it during the sprint.
  • It’s not mandatory to do any prework, but it will help if you know some Markdown.

References

Join us for Tech Writing 101 workshop at Write the Docs Australia

As part of the Write the Docs Australia 2019 conference, we’re running a two-hour workshop on the principles and techniques of technical writing. You don’t need to be a tech writer to qualify for the workshop, and Write the Docs welcomes anyone who’s interested in technical documentation. Do come and join us!

Tech Writing 101 is a class developed by tech writers at Google to train engineers and others in the principles of effective technical writing. Everyone at Write the Docs Australia Conference 2019 is welcome. For tech writers and others, the class offers an interactive, discussion-filled approach to learning tech writing patterns.

Date and venue

  • Date and time of the workshop: Friday 15 November at 2.45pm.
  • Location: Justice and Police Museum, Sydney. See the map and venue details.

Workshop details

Tech Writing 101 consists of two parts:

  • Pre-work that you should read before the class. Don’t be put off! The pre-work is full of interesting patterns and points for discussion: https://github.com/LisaFC/tw101-reading
  • A two-hour workshop at Write the Docs Australia 2019, where you integrate the principles from the pre-work into your writing practices. The interactive format of this class has proved to be an effective way of learning the material. And it’s fun!

Training the trainers: Google is planning to make the course material available for tech writers all over the world who want to run the workshop. You can take part in the workshop purely as a participant, or you can have a dual purpose in mind: take part in the workshop and at the same time observe the facilitator and assistants, with a view to running the workshop yourself sometime in the future.

Your workshop hosts are three tech writers from Google:

Prerequisites

  • Read through the prework, so that you can enjoy discussing it during the class:
    https://github.com/LisaFC/tw101-reading
  • Bring a laptop with WiFi capabilities. (A laptop is not essential, but it makes things easier. During the workshop you’ll write some short pieces and review the work of other participants. Having a keyboard and editing facilities helps.)

Cost (included in conference admission fee)

The workshop fee is included in your conference admission. There is no extra charge for the workshop.

Who’s welcome

All conference attendees are welcome! The workshop is intended for people who want to understand the basic principles and techniques of technical writing. The workshop material is tailored for engineers, and is suitable for new technical writers, editors, UX designers, and anyone else interested in technical communication.

Experienced technical writers are sure to enjoy the content too, both as a refresher and as a reminder of how debatable technical language can be.

The workshop format is highly discursive and entertaining. You’ll find yourself debating the principles as you apply them. The material is designed to stimulate such discussions.

The venue?

No, this is not a picture of the venue. It’s just a building that I found interesting on a recent trip to Prague. I’m also rather taken with the reflections in the windows, showing the windows of the opposite building for a nice Renaissance effect:

%d bloggers like this: