Why We Can’t Judge Relevance



Yesterday, I argued that Microsoft’s search engine update, which included highly touted relevance improvements, did not and will not ultimately improve their fortunes.

To state things a bit more strongly: I think that relevance cannot be a selling point for a search engine, and not just because Live’s update was just catching them up to the level of many other popular search engines. In fact, I think that it’s hard for any of us to truly evaluate “relevance” in results.

There are a number of reasons that I think this is so. First and probably foremost, search engines try (as much any computer can “try”) to understand user intent, but they aren’t all that great at it. While they’ve been preprogrammed to return a specific type of results page for queries they recognize (like music pages at Ask, Yahoo and Google), they can’t automatically parse and understand what you’re looking for.

For example, a search for [beach park virginia] returned nothing I saw as relevant. While I knew the intent behind that phrase, Google could only find pages that had all of the same words there. Whose fault is that?

Sometimes search engines have a bit of help in picking up on user intent. That’s the premise behind personalization: gather enough data about a person’s search habits and you’ll be able to understand what it is they want when they type in [oneida].

But even personalized systems aren’t perfect. Yesterday, perhaps I was looking for flatware; today, I might be researching New York Indian tribes. Tomorrow, religious collective movements. This is part of the reason why there is a limit to how personalized Google has made its personalized results.

Like personalized results, our perception of relevance is subjective. As Phillip Lenssen said on a comment on a Google Operating System post last month:

A 51% “success rate” could mean Google is “only” the best 51% of the time, or it could mean Google returns exactly the right result for the tastes of 51% of the people, and that the other people prefer other types of results.

As we know, the brand associated with a SERP (and the image of the brand) has a great affect on how relevant we think the results are—even when the results are, in fact, exactly the same.

Add to this the fact that the snippets on the page aren’t long enough or detailed enough for us to really tell what we’re clicking through to. A site could be totally on-point for my query, but if it requires me to register, forces music upon me, features a horrific amount of ads or is simply completely illegible, I won’t be able to consider it “relevant.” (And I will run far, far away.)

Once we get to a site, design and other “irrelevant” factors affect our perception of a site, making it difficult to isolate ‘relevance’ alone as a cause for someone to hit the ‘back’ button. And who’s to say that hitting the ‘back’ button means a site is irrelevant, anyway? How many times have you gotten the information you needed and were done with a site?

Objective measures of relevance, on the other hand, are made in a vacuum. They are far outside the real world and our realm of experience. In an objective measure of relevance, the tester types in a query, which they probably didn’t choose. [Apple], perhaps.

And then tester judged how relevant the results are. But “relevance” here isn’t determined by what the searcher really wanted when they typed in the query: it’s what the research team decided was the “right” answer when you ask a search engine “[Apple]?” If their definition doesn’t include Braeburns, suddenly the search engine is wrong.

One of my college professors called this problem with research “The Utterly Boring World.” In this world, The man bit the sandwich is a perfectly fine construct, while *The sandwich bit the man is ungrammatical because it is nonsensical. But there is a place for nonsense in the real world—and a place for Braeburns on a SERP, even if that wasn’t what you were looking for. It might be exactly what someone else wanted.

  • Pingback: » The Search For Relevance And It’s True Meaning Search Engine Optimization Journal

  • francois Covillard

    it’s certainly impossible to “judge” Relevance but possible to estimate it; First the question has a meaning only if you say : relevance to a subject for someone.Then this is a statistical matter . To make a long story short this is the question of democracy (the basic Google assumption) and knowledge. Improving relevance means to give different answers to different people. Either you try to make a profile of any user ( spying on him) or you ask any user to profile himself (by making specialised search engines for instance).

  • http://bushidoblog.com.ar Zen

    It truly is a pickle. Even if you use the personalized results based on your Web History, nothing assures you that one day you’ll be searching for the exact opposite and not find it because of your trends.

  • http://bushidoblog.com.ar Zen

    Totally off-topic:

    I recently followed in your footsteps (actually thanks to a friend) and signed up on Facebook. I finally see what all the fuzz is about! :D
    Kudos on your love for Facebook. May it never die down.

    P.S. If you want, you can add me: Dario Manoukian :)

  • http://www.thevanblog.com Steven Bradley

    Jordan I mentioned it in the previous post, but I think it’s not necessarily about relevance, but rather the perception of relevance and I do think that perception is a selling point.

    It might not be a selling point in the sense of telling me that your results are more relevant, but if you can’t find what you want at any search engine how long are you going to use that search engine?

    Most people use Google. As long as Google can deliver results that are good enough those people will continue to use Google. If the results lose relevancy for some then those some will try and then use another search engine.

    As long as the good enough results are returned the perception will be that the search engine is delivering relevant results.

    Now I agree completely that we can’t really judge relevance. Relevance is subjective and changes from individual to individual and moment to moment.

  • http://targeted.at nmw

    You really need to choose your search engines more wisely (I’ve been arguing this for several years already — see e.g. my exchange with Vint Cerf: http://www.circleid.com/posts/vint_cerf_keynote_domain_roundtable/#2031 about a year and a half ago)

    As an example, when I want to download some software, I find http://download.com works quite well. Their search engine presents far better results than “one-size-fits-all” (or so called “universal”) engines (e.g. Google).

    If you simply choose the appropriate engine, you may indeed find better results.

    Now I don’t know what [beach park virginia] refers to, but I tried searching on virgina.org for beach park and the top result was titled “Gloucester Point Beach Park: Virginia Is For Lovers” (don’t know if that’s good or bad) — it really still is quite early in the “search” game, and perhaps you shouldn’t set your evaluation standards too high right from the outset.

    There are quintessentially two types of searches:

    1. known item search (e.g. “Jordan McCollum” or “TimeWarner” or “Mitsubishi Lancer Evolution” — see also http://battellemedia.com/archives/004084.php )

    2. “topical” search — for example “downloads”, “hotels”, “shopping”, “cars”, etc.

    Generally, more focused search engines are far better at returning relevant results — primarily because such focus enables the engines to pay more attention to details that “one-size-fits-all” engines are not able to pay attention to (because of their generalized nature).

    I hope these “insights” will empower you (and other people) to better choose the appropriate search engine for your needs — choosing the correct engine is, indeed, one of the most crucial elements in the entire search process.

    :) nmw

  • http://www.jordankasteler.com/utah-seo-pro-blog/ Utah Search Engine Optimization

    I think the increasingly popular natural language search engines like: haiku.com and powerset.com are going to send a strong message to Google about relevance in search queries.

  • http://www.jordankasteler.com/utah-seo-pro-blog/ Utah Search Engine Optimization

    Correction: hakia.com not haiku.com :)

  • Jordan McCollum

    @nmw—Actually, you’ve proven my point again. As you say, “I don’t know what [beach park virginia] refers to, but I tried searching on virgina.org [sic] for beach park and the top result was titled “Gloucester Point Beach Park: Virginia Is For Lovers” (don’t know if that’s good or bad).” Exactly. The problem there, as I said in the post, is that you (and the search engine) can’t tell what that search is for (and neither can a search engine—any search engine).

    I’m not convinced that relevance is entirely dependent on choosing the right search engine. Frankly virginia.org isn’t a search engine. It just shows Google’s results. (Unfortunately, they’re still not relevant to my intent behind that cryptic query.) It’s no secret that those are Google’s results on the page—it says “Powered by Google” right there.

    Download.com is the content site that happens to have internal search, not a search engine. If you didn’t know about download.com, what would you do?

    Perhaps instead of telling me that “you need to choose your search engines more wisely,” we need to refresh our memory about what a true search engine is. Personally, I’d rather have a single, real search engine I could turn to with nearly any question than having to find and memorize half a dozen or more specialized sites.

  • Jordan McCollum

    @Steven—Sorry I didn’t respond to your comment on the first post; I was out of town last week (and on . . . DIAL UP. Horrors.).

    I agree that it’s the perception of relevance that’s important—because as I say here, we can’t really judge relevance anyway.

    I believe that this passage from the previous post speaks to your point about Google:
    “Even the most ardent acolyte of Googlism would one day have to turn away if Google began returning aphasic results (you type in [puppies] and get information on chrysanthemums). Search engines should constantly work to improve the relevance of their results—behind the scenes.”

  • http://most-popular.info nmw

    Hi Jordan,

    it hadn’t occurred to me that virginia.org uses Google as it’s site search provider — but I don’t think that matters, either: If it were something about Virginia, I would search this site (maybe it is, maybe it isn’t — only you can be the judge of what it is you were searching for).

    If I were searching for downloads and I found the information at download.com to be insufficient, I might try download.net or download.org — but presently it is still the case that (due to supply and demand), the highested value is placed on .com names (so that is why it is reasonable to try them first). In future years, the TLD’s jurisdication (and thereby the laws governing that TLD) will become a more and more significant factor in “choosing domains” wisely.

    What methods people use to search for information may be different than your preferred method. If you feel that only certain technologies qualify as “search engines”, then you may be limiting your options too much. You do not need to “memorize” anything — a command of basic English vocabulary (especially the most popular “keywords” used in commerce) ought to suffice.

    If you limit your scope to “one-size-fits-all” search engines, you will probably have to sift and winnow through alot of irrelevant “false drops” (as they are called in the jargon of “information science”). Information scientists have been building indexing services for many more decades than Google has, and selecting the correct index for searching is a rather fundamental step in information retrieval — and is much more significant than precisely which algorithms are applied.

    That said, I do acknowledge that a “handy” encyclopedia is a neat reference tool for quick answers to simple questions.

    ps/BTW: AFAIK, download.com DOES primarily index software available at other sites

  • Jordan McCollum

    You’re right about download.com; I’d forgotten that. However, I completely disagree that we can or should even expect to be able to type in a single keyword with a TLD and expect to find there a specialized search engine perfectly tailored to our needs.

    While information scientists (and information retrieval scientists) have been working on this for decades, I’m gonna guess that they’re not being employed by virginia.org (or virginia.com, which you say should be more valuable to we searchers).

    I just don’t know what to say to an argument that says that I should go to any site other than Google/Yahoo/MSN for search results, even if the results come directly from those search engines. I’m gonna keep getting my search results from the source rather than the exact same search results on a branded website. I’d rather go to Google than try half a dozen different keyword+TLDs to find some site that just going to give me Google results.

    Thanks for your input.

  • http://leaderboards.info nmw

    Well, your mileage may vary — and like I said earlier: we’re still at a quite early stage in this business. I ALSO wonder whether people prefer to do their Christmas shopping at http://shopping.com or at http://gifts.com or whether they will hope to find something by using Google — this might be interesting to track over time (and also among various “user groups” — such as “students with alot of time to sift & winnow through lists of thousands of results” vs. “executives who want to cut to the chase and get useful results right away”, etc.)

    At any rate, AFAIK Google is not a “source” of information (but rather they are *supposed* to be the “source” of algorithmic code that makes “judgments” regarding the how “popular” certain webpages are — but ever since the “miserable failure” fiasco they are now seen as an UNRELIABLE source of such judgments… prompting the knowledgeable searcher to ask: “what have they done for me LATELY?” ;)

    Purely link-based search has been returning increasingly poor results for quite some time already — it really only works for brand-names, trademarks and similar “known item” searches (as I described in my first response above [ http://www.marketingpilgrim.com/2007/11/why-we-can%e2%80%99t-judge-relevance.html#comment-37186 ] ).

    And I applaud your integrity in seeking to go to the source! Isn’t it also true that Google is in somewhat of a tussle WRT copyright?

    I think it would be a good trend if not only the sources of information but also the “selective dissemination of information” (you may want to go back a couple decades to draw that skeleton out of the closet ;) were attributed with more transparency. Is Google undertaking anything to clarify how its so-called algorithms and/or value judgments are made?