Why you should not rely on bug reports

After building websites for a few years, I’ve reached a very important conclusion:  you cannot rely on bugreports to find major bugs and flaws in your product.  Think about it, what do you do if a website has a major bug? e.g. the registration form doesn’t work? You will just go to a competitor, unless you have some special interest in helping the creator or you really really really need to use it.
The same goes for missing features or conceptual errors: users just leave if they do not like what they experience.

Users that send you feedback, only do so if they already committed to your product. Maybe they’ve already paid, maybe they just entered loads of content or maybe they just like you.  However, this will only get you so far: in most cases this will be exceptions, obscure bugs or obvious features you just didn’t get done yet.

So what do you need to do?

  • Create a list of all important user stories in your site.  For PicturePush I would write stuff down like:  signup, login, create an album, upload a photo to an album, delete the photo, delete the album,  change my profile, delete my account.
  • Test all of it in ALL major browsers. No matter how much you think you’ve made everything cross browser,  there are always subtile differences that sneak in when you add new features or fix stuff.
  • REPEAT this as often as possible. Fixing bugs and adding features, you are very likely to create new bugs, clutter and/or unclearity.
  • Measure if you can. Using Google Analytics or other tools you are able to track whether users reach certain goals. This will allow you to see if your users succeed in the most critical tasks on your website.

Furthermore you can encourage you users to send you feedback:

  • Offer rewards for feedback. You could give away a few months discount.
  • Make it easy for users to delete their account and have an easy option to indicate why they want to leave. I suggest having radio buttons for those that are in a hurry and an aditional text field for those that want to help you even more.
  • Respond quickly to all questions and bug reports, if you can fix their problem quickly they will not only become loyal customers, they will keep contacting you with feedback.

One Response to “Why you should not rely on bug reports”

  1. 5 Reasons to delay your product launch | Michiel . van Vlaardingen Says:

    [...] 5. You did not reviewed and tested the code. Especially if you have outsourced development, I think you need to closely examine the quality of your product internals. If you can’t do it yourself, hire someone.  Even the best developers make mistakes and take shortcuts once in a while.  Even if you have tested everything, you are still going to find lots of bugs and flaws in the first weeks, so the more you find yourself, the sooner you will get real feedback.      If you believe you can rely on bugreports by users:  you can’t. [...]

Leave a Reply