Hey Folks! The Race is Under the Water.
The picture I put here is an iceberg. The tip of the iceberg is much smaller than the bottom part as everyone knows. This is a commonly used analogy to illustrate the difference between POPULAR and LONG-TAIL queries.
In scientific terms, the difference in magnitude between the POPULAR and LONG-TAIL is actually much larger. Long-tail is more like a black-hole, seemingly infinite, dark, cold, and merciless against popularity algorithms. These are the unpopular, longer than usual, complex, unique, and personal queries. These are the ones that need precision, accuracy, and relevancy.
You know this. We at hakia know this. Silicon Valley knows this. Tech writers know this. VCs know this. Everyone in the search industry is supposed to know this. Very well.. Then what’s the problem?
The problem is this: When a new search engine is mentioned today, with the claim of better search experience, why is there no mention of how they will handle the LONG-TAIL? Why is there no mention of how Google, Yahoo, MSN will handle LONG-TAIL in the future with their popularity-based arsenal of algorithms? Then how can a human-labor based search engine (like Wikia or Mahalo) be viewed as progress in search with the obvious impossibility of handling the LONG-TAIL?
Hey Folks, the race is under the water, the bottom of the iceberg.
I will continue to welcome all those new advances in fancy user interfaces, visual tools, and so forth. But, I raise my hat to some other newcomers who are talking about the LONG-TAIL. I want to see new mathematical methods, learning algorithms, nonlinear mapping techniques, and linguistic/semantic approaches coming in.
Most of all, I want to see the end of this clear and present danger of “avoiding the LONG-TAIL talk.” Progress requires open communication about the dark areas, not about the shining tip of the iceberg.
June 7th, 2007 at 3:51 pm
Great blog!
June 7th, 2007 at 4:35 pm
Hey Amitava, Thanks!
June 7th, 2007 at 7:44 pm
I wouldn’t look at this as one or the other, but rather both.
Humans with machines will always do a better job than those same machines on their own–that’s super obvious I know. Of course, humans don’t scale–which is also super obvious.
What we are doing at Mahalo is just the fat tail. We give our long-tail searches to Google–which does the best job at long-tail searches right now (again, obvious I know). Our hope is that folks like Hakia and Powerset (as well as Google, Yahoo, Ask, and Microsoft) keep making long tail searches better. We’re gonna always use the long-tail services to curate our human searches, and some day we might consider switching from Google to another service (it would have to be better than Google of course).
Also, the long tail searches are not as frustrating for users right now as the fat tail searches because there is more money to be made in the fat tail. As such there is more gaming and index spam in the fat tail. So, right now “Paris Hotels” and “Flatpanel TV” are brutal on machine search due to index spam, but finding the person you went to college with is easy (since there is no financial incentive to game that term) and spam free.
Mahalo for thinking of us and for the feedback!
Good luck in the cold, dark water… we’re gonna stay up top.
best jason
June 7th, 2007 at 8:08 pm
We raise our hats (or should I say scuba masks) to you as well.
June 9th, 2007 at 3:05 am
hi
i think what ever the race is, only the fittest will survive by attracting the users who know what they are doing .
thx