We illustrate 10 points below that define semantic search using our online demo where we compared hakia’s enterprise search system with Pubmed’s search engine, side by side, QDEXing 20 million documents on Pubmed.

1- Handling morphological variations
A semantic search engine is expected to handle all morphological variations (like tenses, plurals, ect.) on a consistent basis. In other words, the results should not change whether you type “improve, improves, improving, improved, improvement”. The example query “improving quality of life” illustrates that hakia results contain morphological variations of the query.

2- Handling synonyms with correct senses
A semantic search engine is expected to handle synonyms (cure, heal, treat,.. ect) in the right context and with correct word senses. For example, the word “treat” can mean doing social favors as in trick and treat, which would not be correct in the medical sense. The example query “is there a cure for ALS” shows that hakia brings results with synonyms with the correct senses. The level of sense disambiguation in a semantic search engine is a sign of its progress.

3- Handling generalizations
A semantic search engine is expected to handle generalizations (disease = GERD, ALS, AIDS, etc.) where the user’s query is expressed in generalized form and the result is expected to be specific. The example query “Which disease has the symptom of coughing?” brings a result set in hakia such that GERD is recognized by the system as the specific answer.

4- Handling concept matching
Perhaps the most challenging functionality among all, a semantic search engine is expected to recognize concepts and bring relevant results (political instability = insurgency, unrest, etc.) Usualy, the depth of this capability is increased in verticals of operation, and it would be unrealistic to expect coverage in all subjects under the sun. The example query “political instability” brings a result set in hakia including concept matching.

5- Handling knowledge matching
Very similar to the previous item, a semantic search engine is expected to have embedded knowledge and use it to bring relevant results (swine flu = H1N1, flu=influenza.) Knowledge match and concept match are similar in principle, yet different in practice in the way the capability is acquired. The example query “swine flu virus” brings a result set in hakia where these kind of matches are visible.

6- Handling natural language queries and questions
A semantic search engine is expected to respond sensibly when the query is in a question form (what, where, how, why, etc.) Note that a “search engine” is different than a “question answering” system. Search engine’s main task is to rank search results in the most logical and relevant manner whereas a question answering system may produce a single extracted entity. The example query “how fast is swine flu spreading?” brings a result set in hakia to shed light to this capability.

7- Ability to point to uninterrupted paragraph and the most relevant sentence
Unlike conventional search engines where a query points to documents, semantic search is expected to do much better. A query must point not only to documents but also to relevant sections of them. This eliminates 2nd search where the user is supposed to open the documents to find the relevant sections. The previous example query “how fast is swine flu spreading?” shows this capability as displayed below.

Semantic Search

8- Ability to enter queries freely, no special formats like quotes, or Boolean operators.
When entering a query, special format requirements are becoming a thing of the past even with today’s non-semantic search engines. These formats perform gross approximations to substitute meaning match, and are signs that unveil the underlying weaknesses of the search technology.

9- Ability to operate without relying on statistics, user behavior, and other artificial means
A semantic search engine is expected to bring relevant results by analyzing the content of a page (or document), its source, authors, and the credibility of the results in response to a query. Relying on link referrals, user behavior/tagging, and other artificial means may produce good results when such data is available, but are outside the realm of semantic search. By not relying on artificial input, semantic search technology is more universal, applicable to any situation especially to enterprise documents and real-time content where such data does not exist.

10- Ability to detect its own performance
When there is no semantic content analysis in a search algorithm, relevancy scores refer to artificial measurements, like how popular the page is. A semantic search engine is expected to produce a relevancy score that reflects the degree of meaning match. This capability provides flexibility for the developers to apply meaning thresholds. Accordingly, the search engine can understand its poor performance to automatically flag areas of improvement that is needed.

In our experience, these 10 points make search a semantic search, and it requires an entire new infrastructure built from ground up. Being able to implement some of these capabilities at full capacity is rare and often unnecessary as it would require tremendous resources. A full capacity semantic search is more feasible in application to vertical topics, especially when embedded knowledge and concept coverage can be attained at a reasonable cost. Focusing on the delivery of concentrated semantic capabilities in verticals is our new strategy in enterprise search. More on this coming soon.

We worked on two elements of progress: (1) automation and (2) relevancy. In both cases, semantic technology is the enabler.

On the automation front, the new hakia.com brings 10 full sets of search results with a single click. You can see the quick progress as the segments come in. These search result segments include Web, Galleries, Credible sources, Pubmed, News, Blogs, Twitter, Wikipedia, Images, and Videos. (Twitter and Wikipedia will be available next week.)

Instead of displaying blurbs from such segments, which is a common practice today, we thought the user should have the full result set in one click, available to him/her for each search.

Although the process of displaying 10 segments may look slow, it is faster than doing 10 searches seperately using any search engine. Furthermore, the increased bandwidth and faster CPUs will make this step instantaneous in the near future.

For those minimalists, the SERP has accordion buttons (little triangles). You can chose what to view and what to hide by opening or closing the segments, as shown below. Your preferences are remembered next time you search, or visit hakia.com.

We believe that the future of search will shift from the domination of a single recepie to the presentation of different segments, almost like restaurants having different menus. Automation is the key for progress in this direction.

On the relevancy front, the relevancy of search results is elevated via our semantic technology at various levels depending on the segment. While Galleries have the highest level of semantic treatment, Credible, Pubmed, News, and Blogs have moderate levels of semantic treatment. All these segments are QDEXed content. The remaining segments receive light level of semantic treatment, mostly on-the-fly, via our SemanticRank algorithm.

At hakia, we are also working on exciting real-time and enterprise search products where the impact of semantic technology is most visible. Stay tuned and expect related announcements in coming weeks.

The Realities of the Web

We have accomplished two important innovations in building the CO. First, the development of concepts and lexicons followed a strict guideline of the realities of Web operations. What were these realities? Most of the search queries on the Web reflect a single dimension of intent, almost exclusively relevant to commercial topics. Note that the interpretation of “commercial topics” must be taken in the broadest sense possible. For example, if you were looking for “the benefits of foot massage” or “the director of the movie Last Emperor” your queries fall into the same commercial pattern. One particular distinction of the commercial pattern is that they come in short packages including a name (onomasticon), or always referring to something sold, bought, watched, heard, etc.

In contrast, many ontologies (if not all) that have been built to date, or claimed so, are focused on the use of language in the general sense, but not in the sense of commercial patterns on the Web. Therefore, their usefulness when tackling the Web search queries is greatly compromised, sometimes to the point of absolute failure. If such an ontology could disambiguate a dozen of different senses of the word “kill”, it would be sad news that the last 100,000 queries in the search logs did not include a single occurrence of the word “kill”. Like drowning in 2 inches-deep water, such ontologies will not utilize their disambiguation skills nearly 80% of the queries because the queries include nothing but onomasticons and/or they are too short (under-articulated).

The Sequence Approach

The second innovation used in the CO is the use of sequences instead of single words. A single word, like “kill”, is the most ambiguous state of information and is hardly used in human communication without a strong underlying/implied context. As a result, building a natural language processing (NLP) systems by taking single word as the unit of computation is an invitation for disaster.

In contrast, word sequences (2 or more words) are inherently safe and highly descriptive. Take “road kill”, for example. This sequence describes a corpse of an animal killed on the road by a passing vehicle. If a language processing system takes the sequences as unit of computation, 99% of the ambiguity problem vanishes. There is no need to process the word “kill” and “road” separately, trace their senses, and locate convergence to identify the meaning of “road kill” if you can just take the sequence “road kill” itself as your unit of computation for mapping. This is depicted below:

Note the number of traces required in a conventional ontology approach compared to the sequence approach. The sequence approach requires a lot of data storage space (which is dirt cheap) whereas the conventional ontology approach requires a lot of CPU for a simple mapping task (which is expensive). But the bad news does not stop there. The trace routes in conventional ontology requires manual work (impossible to automate) whereas sequence-based ontology can be easily built via automation.

I realize only a handful of people will understand the second point above. Nevertheless, the scalability and performance of the end product will speak for itself when we put the testing platform on-line.

Usage of the Commercial Ontology

The immediate use of the CO is related to search queries, or document characterizations, that are not tied to any advertising in conventional systems. This unrecognized domain of search queries and characterizations means loss of revenue. hakia’s CO is designed to fill this gap. For example, if the search query or page characterization is “beat generation” the CO can map it to “literature” on the fly. As a result, systems using the CO will have much deeper understanding of the incoming terms, thus will be able to recognize the underlying intent beyond the face value of the words. The same capability can be used in a number of places other than advertising with the same effect.

Stay tuned for the release of the first version of our commercial ontology.

hakia Galleries received industry accolades after their formal introduction in 2007. Our goal has always been to take search beyond 10 blue links. It was then no surprise when Microsoft invited us to show them the inner workings of the hakia Galleries in July 2008- shortly after their acquisition of Powerset. But it was a huge surprise to recently find out that Microsoft introduced categorized search in Bing. Today we checked out the Bing preview and compared the Bing’s categorized search feature to its inspiration, hakia Galleries.

hakia Galleries provide categorized aspects of search queries. For example, if you are searching for Obama, you can find information about his official site, headline news, images, biography, speeches, and more (see image below). Powered by semantic search, hakia Galleries prove 17 aspects of this query. We save the user time by answering 17 Obama related questions in one search. Compare the hakia Obama gallery with the same search at Bing.com (Bing provides only 7 aspects of this search query).

Let’s look at another example. Search for lung cancer at hakia and Bing. hakia provides the searcher links for the following aspects of this query: Basic Information and FAQ, Image Search, Headline News, Symptoms and Diagnostics, Treatment, Procedures, and Therapy, News, Clinical Trials, Healthcare Facilities and Finding a Physician, Alternative Therapy, For Kids, Research and Statistics, Organizations, Message Boards, and Images. Compare that with Bing’s aspects: articles, symptoms, treatment, prognosis, stages, clinical trials, and images. Look familiar?

As Danny Sullivan put it aptly in his Bing review, “Probably the most significant change is that Bing now organizes search results into categories (gives Obama example). The concept of grouping results also isn’t new. Long known as clustering, you can see it in operation at hakia (see Obama there) or Clusty (again, see Obama there).”

At hakia we could not dream of a marketing budget of $80-100 million. But hey, if you are out there to try Bing as an alternative search engine to Google, give the original categorized search a try at hakia.com (one of Bing’s inspirations!). You can surf the hakia Galleries here: http://gallery.hakia.com/ or try your search at hakia.com when you bing and ding.

CONTEXA provides page-level contextual analysis on-the-fly and outputs keywords that represent the meaning of the page along with their meaning score. CONTEXA is offered as a service and can be integrated into any ad system. RWW has integrated CONTEXA where our system matches the contextual representation of a blog page with sponsors’ requirements on-the-fly to provide relevant ads to RWW readers for a richer experience. The red box in the image below shows this step.

We believe that more relevant contextual ads will bring the return of contextual advertising closer to paid-search levels with the ripple-effect of increased CTR- conversion rates- revenue. CONTEXA is powered by hakia’s semantic core technology. To see how CONTEXA works, you can visit our CONTEXA page.

We had shared with our readers the comparison demo of hakia’s contextual capabilities with that of AdSense and Yahoo in the fall. We did not have a chance to do a comparison with Microsoft’s PubCenter. As we move along with the ReadWriteWeb’s implementation of CONTEXA, we will report about lessons learned and milestones marked.

We are excited to keep the wheels of innovation turning at hakia as our industry has plenty room for improvement. Today, Web users are overwhelmed with the quantity and suffer from the quality of display ads and quickly learn to ignore a good portion of the Web pages they visit. In the long run, the industry’s focus will have shift to increasing ad quality and limiting the supply to increase value. The path to this promise goes through enhancements to both contextual and behavioral ad targeting technologies. We are happy to partner with ReadWriteWeb, a kindred-spirited innovator, for the beginning of a journey to provide more relevant contextual ads .

To learn more about CONTEXA, please contact bdev at hakia.com We are more than happy to set you up with a custom demo.

Last year, over 1.9 million votes were cast last year to select the winners, including hakia in the “Search and Reference” category. To make that a reality once again, please vote for us here!

Thanks to our community of users for supporting the search engine and recognizing the importance of semantic technology for the future of the Internet. We look forward to more progress to come as we near completion of development.

Here is a teaser, the first question.

Take the hakia Quiz now at http://company.hakia.com/quiz/quiz1.html. Enjoy!

An example is shown below for the query “roman invasion of jerusalem” using Google.

With the hakia Scoopbar installed and Highlight button activated (as shown above), you can open long documents and the search result will be located on the page by automatic scrolling and highlighting (as shown below.)

Auto-highlighting is increasingly becoming more important to tackle the 2nd search problem especially for longer documents. hakia team is committed to improve this functionality for the Web searchers.

Note that hakia ScoopBar does not monitor user behavior, does not track Web traffic, and comes with uninstall option. Give it a try and let us know your opinion.

In the not-so-distant future, students will be able to graduate from high school without ever touching a book. Twenty years ago, they could graduate from high school without ever using a computer. In only a few decades, computer technology and the Internet have transformed the core principles of information, knowledge, and education.

To read the full article click here.

Project Syndicate is an international association of quality newspapers devoted to bringing distinguished voices from across the world to local audiences everywhere, strengthening the independence
and upgrading their journalistic, editorial, and business capacities.

There have been many trivializing depictions of semantic search and semantic Web in the blogosphere, so much so that I might have developed an allergic reaction reading them. However, Marshall is doing the right thing by provoking us to define this space better.

First of all, what is “semantic data”? I think what this means is “syntactic extraction” as I followed the examples described. The extraction problem by fitting syntactic patterns, sorry to disappoint some of you folks, is really not semantic analysis. Extraction problem has been around many years, and is being implemented all over the market in enterprise (and government) applications.

Take a word pattern “what is the capital of –” or “what is the capital city of –”. Then, obtain a two column list from the Web of the capital cities around the world. After 12 minutes 34 seconds programming, you will have an extraction algorithm (extraction from the query) just as how Google does in these examples… This is not semantic analysis.

One step further, you can sit down and define patterns until the cows come home, and end up with a large library of extraction algorithms. You might scan through Wikipedia to collect data (if you don’t care proper authorship and credibility). Then you will have something useful, no doubt about it. However, these are not to be considered as semantic analyses.

Bruno Haid expressed his concern by using the terminology “structured versus unstructured platform” for the target of extraction. That is still not enough differentiation between syntax versus meaning in my book. For anything to be considered “semantic” there has to be a model of understanding, involving concepts and associations.

I recommend an old article written by George A. Miller on the ambiguity of words which should inspire a thought as to why syntax-only approach cannot replace meaning. We had posted a fun example here following Bill Gates’ vision. An example of semantic parsing was also posted here previously.

The most important question is how to implement semantic analysis in a search engine environment. The examples in Marshall’s post do not come close to any kind of semantic analysis beyond simple extraction operation. Google has not shown any clues to make us think of an actual semantic back-end yet.