Reviewing this article from 2006 from EUI and Jared Spool struck a few key points:

• “Scavenger-hunt tasks work best when you’ve thoroughly researched the types of things people look for on the site. Our tasks came from extensive interviews and field research. Unfortunately, many times, teams just make up their tasks without doing the research. That’s where the problems begin.”
  The act of creating and selecting tasks needs to flow form solid research – so that your assumptions are verified and correct – otherwise UXD practices can easily get off track.
• “We were quick to see that people who had passion for the tasks behaved quite differently than those that didn’t. People with passion demanded more from the content on the site.”
  So true, many tests with “distant” tasks to the user are just quick run-through’s with no emotional investment in them. Finding that sweet spot where the user has interest, vested value and can clearly express their need and goals and get there is ideal. There is a clear sense of what’s worked and what doesn’t – the user has a clear need they want fulfilled.
• “In interview-based tasks, the participants interested are discovered, not assigned. Unlike scavenger-hunt tasks, the test’s facilitator and participant negotiate the tasks during the tests, instead of proceeding down a list of predefined tasks.”
  Not only getting to know the target user groups as a project goal, but also getting to know the particular person right in front of you in testing add value and the best results.
• “With interview-based tasks, participants take us down paths we never expect to go”
  I love this fact! We are so directed by our own understandings, experiences, and mental models. Quality UXD is in the commitment to LISTEN.
• “Terminology emerges to describe user needs in a way we hadn’t previously thought.”
  An open card sort in the context of usability testing – nice.

As always, thanks Jared for sharing.

The challenge in dealing with conformitory bias – “People can reinforce their existing attitudes by selectively collecting new evidence, by interpreting evidence in a biased way or by selectively recalling information from memory.” – is real. Getting “over yourself” and seeking out valid user testing and feedback is a tough call when you’re doing it all yourself, design and testing.

The points that Paul J. Sherman makes in his followup article that strike me most are these:

• Testing is not a Pass/Fail activity – it is a valuable element in the Design Process. Focusing the testing as a part of the path to a valid, usable solution is key.
• Test early, often, and as lo-fi as possible (keeping the initial investment, emotionally as well as in time/cost down).
• Usability testing serves design – use it to improve design.