Archive for April, 2008

Popular votes determine the leaders of our societies as a fair practice of equality and human rights. Popular votes among expert physicians can produce better diagnostics of a medical condition. There are many other cases where popularity, as a method, works and functions well.

But, how well does popularity work for the Web search?

Like the saying “if the only tool is a hammer, everything looks like a nail” popularity ranking is the only perspective available out there, thus every result looks like a nail! There is nothing else to compare.

A closer look, however, shows that popularity can sometimes fail miserably without the need for any comparison. Worse than that, it fails in a hidden way at a much higher rate than what we want to believe. I would like to talk about these issues in this post, and explain why we are doing what we are doing at hakia.

Let’s first look at the validity of popular view. Below are some fun examples of how popular view fails. Obviously, this has no impact on our lives other than making us look stupid. There are so many of them that you can find books published in this area, but let’s just list 3 of them.

Contrary to popular belief:
- Nowhere in the Bible is the fruit eaten by Adam and Eve referred to as an apple
- Thomas Edison did not invent the light bulb
- Seasons are not caused by Earth being closer to the sun in summer than in winter.

Fun right? The next category is not fun at all. The history is full of cases in medicine, science, technology, politics, etc., where popular belief produced deadly consequences.

Contrary to popular belief (in history):
- More than 40,000 women executed in Medieval Europe were not witches.
- The Titanic could sink despite its unsinkable reputation.
- To imitate the Marlboro man was not cool at all.
- Lead piping and asbestos were harmful building materials.
- HIV could spread among heterosexuals.
- There were no WMDs in Iraq to pose an immediate danger.

The lesson to be learned from all these examples is that popular view can be wrong. While some of the misconceptions may be innocent due to lack of scientific data, most of them are byproduct of information manipulation for commercial or political benefit. A careful eye will catch millions of such manipulations still at work today.

Now, let’s switch back to Web search. If search results are ranked and organized by a popularity algorithm, which reflects peoples’ choices as to what is right and relevant, how can you trust this view for important queries in health, finance, law, business, etc.? Can some of these results be commercially-biased, politically-biased, or innocently incorrect? Are there better results that you don’t see? We call this the hidden failure.

This is where we separate ourselves from the current wave of popularity algorithms. The semantic algorithms at hakia are not based on collecting statistics on link referrals, click behavior, or on any other similar measure. Our criteria is quality, which is defined as the combination of credibility, freshness, and relevance by meaning match. It is our vision that this new perspective will benefit the Web searcher by minimizing (if not eliminating) the extra burden of quality assurance. There will be no hidden failures by design.

Note that we haven’t even touched the subject of long-tail which is an inherent technical limitation of popularity algorithms. Long-tail limitations directly contribute to the hidden failure.

For the Web searcher, the only tool is no longer just popularity, and we are working hard to bring quality search as an alternative perspective. Until then, keep challenging hakia BETA to benefit from the ongoing progress and to give us feedback.

We are happy to announce that hakia.com received a 2008 Webware 100 award for “Search and Reference” by Webware, a CNET site. We thank you for your support and votes!

The 2008 Webware 100 awards recognize the best Web 2.0. sites, services and applications. The Web 2.0 user community cast nearly two million votes in an online voting poll which ultimately selected the winners. Finalists for the 2008 Webware 100 Awards were selected by the editors of Webware.

We are proud to be the only New York-based search company and the sole flag bearer of semantic search in this short list. Bear in mind that we are continuously working on our technology to make the hakia products better.

By this award, an important question may have been answered: Is “semantic search” a reality now?

Webware 100 voters think so. We say “it is only the beginning.”

Last night we hosted the April meeting of the NY Semantic Web Meetup. The presenters were Richard Cyganiak from DERI and Dr. Christian Hempelmann, our Chief Scientific Officer.

Richard gave an introduction to the future of open linked data on the World Wide Web. Christian presented the “Search for Meaning” the hakia way and gave a quick intro into our OntoSem technology and what its license package includes. The Semantic Web v.s. Semantic Search discussions were heated and lively. I expected nothing less.

As the pictures show, our office was packed. Some people wanted to see the tomatoes we grow in our 33rd floor terrace, but it is unfortunately not the right season.

Thank you all for coming! We would also like to thank Marco Neuman, the organizer of this dynamic Meetup for building the “semantic community” in New York.

Our recent BETA update stirred interesting conversation on the blogsphere along with questions and scepticism. We welcome all comments and take the feedback from the users to improve our search engine. I want to take this conversation one step further.

When you are searching for any topic on the Web, no matter what, you will find yourself in between the worlds of different views. There will be good and bad things about it written by opposing views, a part of which will have a commercial interest.

When the good and bad are clear, there is no problem. For example, burning coal for energy is good because it is cheap, but it is also bad because it pollutes the environment. The distinction is as clear as it gets.

When the good and bad are not clear, there is uncertainty which is the ugly. What makes uncertainty ugly is the commercial motivation to interpret it one way or another. In medicine, law, finance, business, politics, and many other topics, uncertainty “the ugly” has been the fertilizer to grow cash and personal gain out of misleading. Some success come from using the ugly in a very unsettled way.

If a corporation, say X, is manufacturing vaccinations in the market, and financially sponsoring some independent organizations, say Y, and if you read the benefits of a certain vaccination on the Website of Y, where does this leave you as the consumer? Does it put you in the ugly? There are tons of Xs and Ys in the world involved in medications, treatments, therapies, and what not.

Unlike in the movie, the ugly is not just “bad looks” in Web search. It may cause pain, both mentally and physically. Forming opinions and making decisions by using misleading information can have serious consequences. To avoid it, you have to spend a lot of time trying to assess the legitimacy of the information found.

As a search engine user, you shouldn’t worry about the ugly. You shouldn’t be tracing relationships to assess the legitimacy of information. This is one of the aspects where hakia differentiates itself from the others. Starting a week ago, we are displaying search results at top positions from credible sources recommended by expert librarians. Those are the ugly-free results.

The user can point the mouse on the corner of a search result to see the following details:

We display the full name of the organization, the librarians who vetted the sources, and the date of capturing this search result. We call it the “quality stamp”. If we display a source like Wikipedia, which is not as credible as the others, we show its nature: “User Generated Content”.

Our objective is to be the first general purpose search engine where the results are ranked by quality rather than popularity. As a one-stop destination site, we want to offer the users credible and fresh results in every vertical. This is possible, thanks to our scientists, building semantic algorithms and resources in the back-room.

Semantic search technology enables accurate retrieval of information via concept/meaning match. It is very effective, and perhaps the only method, in application to credible and dynamic content. Popularity algorithms cannot work effectively beyond common queries because most of the credible and dynamic content are statistically flat (infertile). That is how semantics is related to quality as the enabling force.

There are more innovations coming your way from hakia. That’s the good thing. It is taking quite an effort and time. That’s the bad thing. And the ugly, we’ve decided to stand up to it.

And Eli Wallach says: Those who are making comments on hakia’s QDEXing ought to read this page before writing a blog post. He means it!

Every Web search starts with a query. Right? Actually wrong.

Every Web search starts with two queries. One is X. The other one is “who knows X the best?” Because finding X is not enough if the author of that page does not know X himself/herself. This will immediately resonate with you if you ever searched for medical, legal, or financial information for a serious case.

This was called the “credibility” criteria in the old world-order which has progressively vanished in the new age of Internet search engines. You enter X, and get the same “popular” perspective without distinction of credibility. You may recognize some of the sources, but are you an expert yourself about these things?

Ironically, there is a science for this. It is the science of libraries and librarians. That’s their job. They know what is credible, trustworthy, and commercially-unbiased. But how is it possible that none of the search engines today provide you a full-perspective of credibility for the given query? It is because they made their choice earlier while building their algorithms. They chose the popular view. Their result pages are doomed to have a mixed view including junk content that slips in because it is somewhat popular. Without semantics, there is no way to stop that!

With the update today, hakia is now presenting its vision of Quality Search (versus popular search). You will instantly understand what I mean by trying this query:

What are the benefits of aspirin?

We define “quality” search as the one that satisfies three criteria simultaneously. The results must (1) come from credible, trustworthy, and commercially-unbiased sources, (2) represent the most up to date information available, and (3) be relevant to the query.

hakia is now QDEXing (our process of indexing) verticals, starting from Health and Medicine. The choice of the credible sources is taken from the Medical Library Association recommendations. Results coming from these sources are clearly stamped and always positioned at the top. The user of hakia now can trust what he/she finds, because the sources are recommended by experts.

The user of hakia only needs one query in his head, not two. That’s the whole idea.

A complete list of medical sources being QDEXed can be found on our corporate site. Our BETA search engine is still not perfect and we are working on it diligently. But for most health related queries, the Quality Search is now entering our lives.

hakia will continue content acquisition in law, finance, science, and in many other content-rich verticals with the aid of librarians. If you are a librarian, please contact us for collaboration by sending an email to info@hakia.com.

For the past couple of weeks, we have been conducting an informal survey in the Wall Street area of New York City, right near hakia’s headquarters. We’ve asked the following questions:

Have you heard of Google?
Are you satisfied with Google?
Do you wish you had something better than Google?
Have you heard of hakia.com?

After talking to 100 people, the results were as follows:

Have you heard of Google (Yes: 100, No: 0)
Are you satisfied with Google (Yes: 82, No: 18)
Do you wish there was something better than Google (Yes: 25, No: 75)
Have you heard of hakia: (Yes: 2, No: 98)

The results validate an important point – the market needs for a better search. When asked, a significant 25% of our respondents said that they wished they had something better. Some were steadfast in their Google loyalty, yet many hesitated before responding. In fact, quite a few of the “No” respondents stated that they had simply never thought about it.

It can be speculated that the 25% wish for a better search engine corresponds to the professional and/or critical usage of the search engines. In such cases, people must find answers to their unique queries for serious reasons. They are looking for something beyond popular, and most likely something credible and fresh.

It is also very interesting that 25% of all queries asked every day have never been seen before in Google. This is Google’s own declaration. That means these questions are unique, in the long-tail region, and coincide with the 25% wish for better search.

We were surprised to find out that 2 people out of 100 heard about hakia. That must be related to hakia being located in the neighborhood. Because, we have not yet started any serious marketing activities to claim 2% awareness.

Well, for at least 100 people on Wall Street, hakia is now giving them something to think about.