The word gets bandied about so much, but realistically you can have 10 people searching using the same words and yet not be looking for the same things...
How do you measure relevancy? There is a need for some set of criteria - give perspective and also something to measure results against.

Have you ever participated in a taste test where you were suppose to guess if a white cup of cola was either Pepsi or Coke? The point behind the test was to pull marketing and branding out of the equation and look at the raw materials. Is Pepsi better then Coke based on a non biased taste test.

How about white labeling search results from Google, Yahoo!, MSN, and Ask Jeeves and asking a group of individuals which results are better. They will not know which results are from which engines. All they will know are the results.

Wonder which search engine will be deemed as most relevant?