A developer’s guide to user testing
Phase 0: A roadmap for open source teams 🗺️⌗
What is this guide? 🏗️⌗
This guide focuses on informal, DIY, beginner user testing in a process that’s not rigorous but extremely valuable. As a short and practical guide, it helps teams get started, but shouldn’t be used to shift into a career of user research.
Who is it for? 🖇️⌗
We wrote this guide with small (2-20 active contributors or maintainers), open source teams in mind. Even if you don’t have a dedicated designer or researcher on your team, user testing can be achievable (and fun!).
What is user testing? đź‘Ą⌗
User testing is a pillar of UX design. You watch someone use your software or prototype, ask them to “think aloud”, and ask them probing questions when necessary. The aim of user testing is to identify problems and gather information you will need to find solutions. During user testing, you can discover usability issues and bugs, but also learn about your users’ motivations. When you know your user better, you can make better software. User testing takes energy, but it’s worth it because you’ll feel more confident in your code, know your users better, and hopefully save time and energy in the future. You may think you’ve thought of everything, but during a user test you’re likely to have a few “mind blowing” moments that reveal your blindspots. That’s because you’re so close to your work - every once and a while you need someone to look at your software with a fresh perspective to see what you can’t.
User testing doesn’t have to be a big task. Any type of feedback from your users is valuable. If you’re already listening to and collecting feedback from your users - that’s great! Keep doing what you’re doing and when you feel you’re ready, come back to this guide - we’ll still be here!
We compiled user testing FAQs in our interactive companion resource “User testing can be fun”.
How to use this guide 🦮⌗
This guide breaks down the user testing process into three phases: planning, recruiting, testing. Each phase has detailed instructions and considerations, as well as example templates. It’s best to read through the entire guide before getting started so that you’re familiar with the whole process. After user testing, the next step is organizing and analyzing what you learned. See the companion resource, Synthesis for open-source, to learn more about research synthesis.
The flow of starting user testing in 4 steps: Plan, Recruit, Test, Learn.
What You Need đź“ť⌗
You don’t need a lot of resources to do user testing. Support from your team is helpful, but at first you may need to conduct user tests by yourself in order to convince others of the benefits. Openness, humility, and curiosity are important qualities to embrace.
Specifically, it’s good to have:
- Time to prepare, conduct, and reflect on the tests (the amount of time depends on the length and numbers of users tests but we recommend 10-15 hrs of time to plan, conduct, and reflect)
- A place to plan the user tests and take notes, like EtherPad/Riseup Pad Google Docs, or Notion
- Something to test, like your current software, a prototype, or a sketch
- (If possible) Another person to take notes during the test
- (If you’re working remote) A video/audio conferencing tool, like Jitsi or Zoom
Get in to the user testing mindset 🧠⌗
You might be thinking at this point, “I can’t do user testing! I’m not ready!” Don’t worry, this is common and happens to everyone, including people who have been conducting user testing sessions for years. A warm up activity can help you and your team feel more confident. Try a variation of the Five Whys, where a team member plays the role of the user (this is just for practice purposes - don’t write code based on team members’ assumptions!).