As you will find, User Centric consultants frequently publish papers in peer reviewed journals and publications, as well as present data at professional associations domestic and international. We believe that we run our studies in a sound and rigorous manner that is in line with the scientific method.

That said, we acknowledge some may argue about the sample size. Yes, we could have run more participants, but that only increases power. Would 30 have been better? 40? 100? While sample size is really a question of power, a majority believe sample size comes down to “this smells large enough”. But, the problem is that this assumes that the speaker says that these results or scores can be generalized to the population. This is not a question of confidence intervals, but a question of difference. Bing and Google’s scores were significantly different. This is an inferential statistics question whether we can infer from the data that the difference is significant. If we were to run the study 20 times what is the likelihood that we could have found this difference by chance alone (p value of .05).

User Centric did not say that the scores were generalizable, but that the difference between Bing and Google were significant. We are able to make that statement and yes, sample size played a role in the statistical analysis performed.

Now, we certainly welcome discussions on new research questions or even if we think the result is due to the novelty of the Bing design and whether this will fade away. Valid points where we could run a study with experienced users and other permutations. Please forward topics for research questions to me directly.

The result is what it is. Let’s talk about the implications, habituation, usefulness of features, whether the options were copied from Google and improved or made worse.

Rich’s last blog post..Can Bing Level the Search Playing Field?

Matt’s last blog post..Yahoo! Web Analytics 9.5 launches and new Y!WA Facebook Group!

This heat map definitely says top placement is important regardless – the bottom results (organic and paid) are barely viewed.

Jayson’s last blog post..Shame on You New Home Source for Stealing Our Traffic

sam michelson’s last blog post..Evolution of the Internet and what it says about us

Brennan’s last blog post..Protecting Your Company In Local Search

The right column gaze duration results are indeed interesting, but could there be other factors in play? For example, the heatmaps you posted had 8 ads in the right column on the Google SERP, and only 5 ads on the Bing SERP. And, who knows, we might be looking/thinking too hard about these gaze duration studies – the similarities are striking.

Btw, I’ve done usability studies on sites with 3 column layouts – particularly those where the right column is bifurcated – very similar to your blog’s layout Andy. What I learned is that users overwhelmingly ignored the content and links in the far right column.

Tom Kasperski’s last blog post..One small step for?everyone