We announced the launch of SENSEnews.com last week, which constituted a prime example of how search technologies will be used in the future.

SENSEnews aggregates news articles from more than 30,000 sources, one million blogs, and Twitter content. Using its semantic search technology, SENSEnews then detects pre-defined criteria in the captured articles to determine the performance of a company and the value of its stock. It was shown to be highly accurate in valuing companies like Google, Microsoft, and Intel using the same methodology. Now, using this technology, we analyzed the value of Facebook.

During the last twelve months, Facebook’s value has been steadily increasing – from $12 billion to $50 billion – based on reported investments and transactions. According to SENSEnews Index, the value of Facebook had an upward trend in 2010 until September, and then it significantly dropped back to its previous level of June 2010 (as shown in the chart below). This brings up the question: which valuations are correct?

There are two different sets of valuations, one recorded before September 2010 the other recorded after this date. The first set includes a January 2010 valuation of $14 billion (offer to buy on Secondmarket) and a June 2010 valuation of $24 billion (Elevation Partners). They are in agreement with each other when mapped on the SENSEnews Index. The second set includes November, December, and January 2011 valuations of $35 billion, $56 billion, and $50 billion, respectively, the last one by Goldman Sachs. All these valuations after September 2010 do not agree with each other, nor do they agree with the first set, according to the SENSEnews Index.

The turning point detected by the SENSEnews Index in September 2010, which was caused by an increased number of court cases and legal entanglements, suggests that (assuming the first set of valuations are correct) the current value of Facebook is actually around $20 billion, $30 billion less than its latest valuation.

It is interesting to note that Goldman Sachs’ $50 billion valuation was achieved by Facebook in September 2010 according to the SENSEnews Index (as seen in the chart.) It was probably when the actual value assessment was made by Goldman Sachs. However, by the time the investment was closed in January 2011, Facebook’s value had dropped.

“We have seen before that increased level of legal entanglements has negative effect on stock prices like in Google and Microsoft” said Dr. Riza Berkan, the CEO of hakia and co-inventor of SENSEnews. He added “Each court case is like a hand grenade, you never know their future impact on a company’s monetization capabilities.”

The estimated value of private companies like Facebook, Twitter, and others, can be monitored at SENSEnews.com via subscription.

Welcome to the new world of systems (products or services), that are riding on semantic search engines!

When one hears the phrase “search engine” his or her expectation might be to type in a query in a search box, and find the button that says “go” or “search”. With the launch of SENSEnews.com, we are proud to introduce a new service for financial markets that is based on hakia’s semantic search technology, and it has no search box or “go” button. This computing with words, which is the specialty of SENSEnews.com, allows the search engine to run silently in the background.

Articles published in premium or subscription services, in free news sources (over 30,000 news sources), in blogs (over 1 million), in Twitter,… they all are now a part of an advanced computation by SENSEnews. For example, for a given public company, this giant information aggregation includes the following:

- Reported events (revenues, store closings, product launches,…)
- Fundamentals (earnings, stock prices, …)
- Interpretations (how analysts interpret financial state of the company)
- Speculations (what the market thinks about the future of the company)
- Sentiments (how people feel about this company, complaints, appraisals)
- Promotions/demotions
- Global factors affecting the company (economy, disasters,….)

Each component above is detected from the news stream, then an aggregate score is computed to produce positive or negative effect on the performance of the company. This is done at petabytes level for every few minutes. One can only imagine the precision requirement that is necessary to tackle this challenge.

For example, for the company Intel, the result is a chart (SENSEnews chart) as seen below that depicts the performance of Intel in the eyes of the media.

We all know intuitively that the performance of a company, when all factors are evaluated, also reflects itself in its stock value. Comparing the chart above with Intel’s stock price confirms our intuition.

Hence, the agreement seen above validates that the blue chart, the SENSEnews chart, is a significant scientific discovery. About 85% of the Fortune 1000 companies we analyzed have stocks strongly correlated to emerging news in a manner similar to the one above.

The discrepancy between the blue curve and red curve represents the inefficiencies of the market where price movement follows the news curve with some time delay. In reality, there will be no case where these two curves will match 100%. Otherwise, the exact match of these two curves would validate the unreaslitic efficient market hypothesis which assumes that all floating information is completely absorbed by the traders and reflected in the stock price.

The new SENSEnews Stock Indicator takes advantage of this “two-curves” information. A stock will be under-valued if there is an accumulation of good news during which the stock price has not yet responded to it. Conversely, a stock will be over-valued if there is an accumulation of bad news during which the stock price has not yet responded to it.

Several trading strategies using the SENSEnews Stock Indicator have yielded returns higher than that of know indexes like DJIA and S&P 500. This has been one of our proof points.

SENSEnews is the beginning of a new era in financial markets, and we are happy that you are witnessing it with us.

Happy New Year!

During 2010, we have further developed the enterprise functionality of hakia in two verticals: Financial Markets, and Aerospace. These verticals are ideal for semantic technology because (1) content is rich and complex, (2) search criteria are much more demanding than that in a typical Web search, and (3) precision cannot be compromised.

We are launching the financial markets product under a new label called SENSEnews.com in January 2011. Aerospace product will be launched later this year.

Are you ready for something new and unique? How about a new stock indicator that shows whether a stock is over-valued or under-valued only by analyzing the news and social media content? How about a new index that shows the state of an industry based on the emerging news?

For example:

Did you know that Google Stock (GOOG) was $5.88 over-valued the last day of 2010?

The SENSEnews stock indicator, as shown below, illustrates how news (good or bad) is moving (blue chart) in relation to the stock price (red). The stock indicator calculates the unrealized value piled up behind news that has not yet reflected on the price. Although the correlation is usually visible (from the charts) the precision of the Stock Indicator was proven by using the indicator as a trading tool that yielded higher returns than that of DJIA and S&P 500 indexes.

Did you know that Automotive Industry has been on the rise in last 3 months?

Another feature available with SENSEnews is the Indexes, built entirely in the same unique way as the Stock Indicators. The news related to the state of the automotive industry is gathered, analyzed, and scored to produce the blue chart.

Stay tuned for more updates on SENSEnews.com.

With the recent article on ReadWriteWeb, it seems like the old debate is back. Are Google’s squared results coming from a real semantic backbone, or is it a good old entity extraction trick anyone, who is capable of copying and pasting lists, could do?

We illustrate 10 points below that define semantic search using our online demo where we compared hakia’s enterprise search system with Pubmed’s search engine, side by side, QDEXing 20 million documents on Pubmed.

1- Handling morphological variations
A semantic search engine is expected to handle all morphological variations (like tenses, plurals, ect.) on a consistent basis. In other words, the results should not change whether you type “improve, improves, improving, improved, improvement”. The example query “improving quality of life” illustrates that hakia results contain morphological variations of the query.

2- Handling synonyms with correct senses
A semantic search engine is expected to handle synonyms (cure, heal, treat,.. ect) in the right context and with correct word senses. For example, the word “treat” can mean doing social favors as in trick and treat, which would not be correct in the medical sense. The example query “is there a cure for ALS” shows that hakia brings results with synonyms with the correct senses. The level of sense disambiguation in a semantic search engine is a sign of its progress.

3- Handling generalizations
A semantic search engine is expected to handle generalizations (disease = GERD, ALS, AIDS, etc.) where the user’s query is expressed in generalized form and the result is expected to be specific. The example query “Which disease has the symptom of coughing?” brings a result set in hakia such that GERD is recognized by the system as the specific answer.

4- Handling concept matching
Perhaps the most challenging functionality among all, a semantic search engine is expected to recognize concepts and bring relevant results (political instability = insurgency, unrest, etc.) Usualy, the depth of this capability is increased in verticals of operation, and it would be unrealistic to expect coverage in all subjects under the sun. The example query “political instability” brings a result set in hakia including concept matching.

5- Handling knowledge matching
Very similar to the previous item, a semantic search engine is expected to have embedded knowledge and use it to bring relevant results (swine flu = H1N1, flu=influenza.) Knowledge match and concept match are similar in principle, yet different in practice in the way the capability is acquired. The example query “swine flu virus” brings a result set in hakia where these kind of matches are visible.

6- Handling natural language queries and questions
A semantic search engine is expected to respond sensibly when the query is in a question form (what, where, how, why, etc.) Note that a “search engine” is different than a “question answering” system. Search engine’s main task is to rank search results in the most logical and relevant manner whereas a question answering system may produce a single extracted entity. The example query “how fast is swine flu spreading?” brings a result set in hakia to shed light to this capability.

7- Ability to point to uninterrupted paragraph and the most relevant sentence
Unlike conventional search engines where a query points to documents, semantic search is expected to do much better. A query must point not only to documents but also to relevant sections of them. This eliminates 2nd search where the user is supposed to open the documents to find the relevant sections. The previous example query “how fast is swine flu spreading?” shows this capability as displayed below.

Semantic Search

8- Ability to enter queries freely, no special formats like quotes, or Boolean operators.
When entering a query, special format requirements are becoming a thing of the past even with today’s non-semantic search engines. These formats perform gross approximations to substitute meaning match, and are signs that unveil the underlying weaknesses of the search technology.

9- Ability to operate without relying on statistics, user behavior, and other artificial means
A semantic search engine is expected to bring relevant results by analyzing the content of a page (or document), its source, authors, and the credibility of the results in response to a query. Relying on link referrals, user behavior/tagging, and other artificial means may produce good results when such data is available, but are outside the realm of semantic search. By not relying on artificial input, semantic search technology is more universal, applicable to any situation especially to enterprise documents and real-time content where such data does not exist.

10- Ability to detect its own performance
When there is no semantic content analysis in a search algorithm, relevancy scores refer to artificial measurements, like how popular the page is. A semantic search engine is expected to produce a relevancy score that reflects the degree of meaning match. This capability provides flexibility for the developers to apply meaning thresholds. Accordingly, the search engine can understand its poor performance to automatically flag areas of improvement that is needed.

In our experience, these 10 points make search a semantic search, and it requires an entire new infrastructure built from ground up. Being able to implement some of these capabilities at full capacity is rare and often unnecessary as it would require tremendous resources. A full capacity semantic search is more feasible in application to vertical topics, especially when embedded knowledge and concept coverage can be attained at a reasonable cost. Focusing on the delivery of concentrated semantic capabilities in verticals is our new strategy in enterprise search. More on this coming soon.

With today’s update at hakia.com, we are coming out of a period of silence during which we made several updates to our offerings on the Web and in enterprise search.

We worked on two elements of progress: (1) automation and (2) relevancy. In both cases, semantic technology is the enabler.

On the automation front, the new hakia.com brings 10 full sets of search results with a single click. You can see the quick progress as the segments come in. These search result segments include Web, Galleries, Credible sources, Pubmed, News, Blogs, Twitter, Wikipedia, Images, and Videos. (Twitter and Wikipedia will be available next week.)

Instead of displaying blurbs from such segments, which is a common practice today, we thought the user should have the full result set in one click, available to him/her for each search.

Although the process of displaying 10 segments may look slow, it is faster than doing 10 searches seperately using any search engine. Furthermore, the increased bandwidth and faster CPUs will make this step instantaneous in the near future.

For those minimalists, the SERP has accordion buttons (little triangles). You can chose what to view and what to hide by opening or closing the segments, as shown below. Your preferences are remembered next time you search, or visit hakia.com.

We believe that the future of search will shift from the domination of a single recepie to the presentation of different segments, almost like restaurants having different menus. Automation is the key for progress in this direction.

On the relevancy front, the relevancy of search results is elevated via our semantic technology at various levels depending on the segment. While Galleries have the highest level of semantic treatment, Credible, Pubmed, News, and Blogs have moderate levels of semantic treatment. All these segments are QDEXed content. The remaining segments receive light level of semantic treatment, mostly on-the-fly, via our SemanticRank algorithm.

At hakia, we are also working on exciting real-time and enterprise search products where the impact of semantic technology is most visible. Stay tuned and expect related announcements in coming weeks.

2009 was a busy year at hakia. We hatched new ideas and started to worked on new products. Stay tuned for our upcoming announcements.

2010 will be an interesting year for search as Bing-Yahoo! Partnership will become effective and innovation from start-ups will continue.

We would like to invite you to an experiment we are running for fun. Please welcome nobrandsearch.com ! NoBrandSearch is an experiment to compare the core competencies of the search engines in the market. It may surprise you to see how your favorite search engine compares to other search engines in a blind test.

This is how it works: Each time you enter a query, each side of the screen is randomly selected from the four search engines (Google, Bing, Yahoo! and hakia) in the market. Their brands and special features are screened out. All you have to do is to decide which set of results is better.

When you click to vote, the aggregate votes are immediately shown between the two search engines on the screen. The overall results will soon be displayed at http://nobrandsearch.com

We encourage you to test the search engines with your complex and longer queries, that’s where the challenge starts!

Join the fun & shall the search-off begin!

Since the launch of the contextual link advertising product on ReadwriteWeb, we at Hakia have been anxious to see the results and evaluate the success of our contextual advertising product, Contexa. Contexa system matches the context of a blog post with a sponsor’s criteria using proprietary semantic technology to deliver relevant ads to the reader. Participating ReadWriteWeb sponsors have provided the contextual engine with up to three “trigger phrases” that define their business. As a reader of the blog, you may have seen the product’s implementation at the bottom of certain blog entries, as shown below. You can see another example here.

As we began this exciting journey, the ReadWriteWeb team defined its objective as follows:

1. “To offer value to our readers by providing advertising links in the context of what they are reading, links that would therefore more likely be of interest to them, and
2. To offer a higher level of engagement to our advertisers, resulting in both more branding impressions and more click-throughs.”

The preliminary aggregated statistics of the six participating sponsors (excluding hakia), covering a 40-day period, demonstrate that the Contexa system has met ReadWriteWeb’s objectives:

- The Contexa system increased ad clicks by 14% (i.e. advertiser received, on average, 14% more ad clicks).
- The click-through rate (CTR) for Contexa was more than twice that of ReadWriteWeb’s 125 x 125 banner ads.

We decided to turn the tables and interview Bernard Lunn, ReadWriteWeb’s COO and Feature Writer for his feedback! Hakia’s own COO, Melek Pulatkonak, poses the questions.

Melek: Bernard, we have been working together on the ReadWriteWeb contextual link advertising system for a while. Has the system met your expectations?

Bernard: Yes, it has. We wanted to see if it would generate a meaningful uplift in CTR, and it has.

Melek: What feedback have you received from advertisers? And what would you recommend to participating advertisers going forward?

Bernard: Advertisers have to get the traffic–relevance balance right. You can drive a lot of clicks with a hot term – something we are writing about a lot – but if the relevance is low, the advertiser won’t get good conversions. As with any new type of advertising, an art and science emerges over time. People know how to buy search terms on Google, but this is a bit different. I think we need to get better at creating more of a feedback loop (e.g. stats on how different terms have performed) so that advertisers can tune their keyword selection accordingly. Each advertiser has different needs and knows its market intimately, so it is best positioned to decide what works and how to tune its selection.

Melek: What’s next? What is your vision?

Bernard: For this first phase, we provided Contexa to our long-term sponsors. In the next phase, we want to offer Contexa as a standalone offering, so that advertisers can purchase keywords (or trigger phrases) directly on ReadWriteWeb. This will be an entry-level self-service advertising option that many smaller startups have requested.

Melek: Anything else you would like to add?

Bernard: Context matters to engagement. That is an obvious statement, but doing it right has been hard, and the opportunities for bloggers to offer ads that engage readers well and offer them value have been limited. Contexa is a good step in this important journey.

We thank the ReadWriteWeb team for working with us closely to create a new contextual ad system for blogs and other publishers. To learn more about Contexa, please contact us at bdev@hakia.com

Perhaps the world’s first, we are proud to announce our upcoming Commercial Ontology (CO). What is a commercial ontology? If you asked this question you have just touched on an important distinction: fantasy versus reality. In the context of World Wide Web, the CO is the realistic version of an ontology for the reasons explained below.

The Realities of the Web

We have accomplished two important innovations in building the CO. First, the development of concepts and lexicons followed a strict guideline of the realities of Web operations. What were these realities? Most of the search queries on the Web reflect a single dimension of intent, almost exclusively relevant to commercial topics. Note that the interpretation of “commercial topics” must be taken in the broadest sense possible. For example, if you were looking for “the benefits of foot massage” or “the director of the movie Last Emperor” your queries fall into the same commercial pattern. One particular distinction of the commercial pattern is that they come in short packages including a name (onomasticon), or always referring to something sold, bought, watched, heard, etc.

In contrast, many ontologies (if not all) that have been built to date, or claimed so, are focused on the use of language in the general sense, but not in the sense of commercial patterns on the Web. Therefore, their usefulness when tackling the Web search queries is greatly compromised, sometimes to the point of absolute failure. If such an ontology could disambiguate a dozen of different senses of the word “kill”, it would be sad news that the last 100,000 queries in the search logs did not include a single occurrence of the word “kill”. Like drowning in 2 inches-deep water, such ontologies will not utilize their disambiguation skills nearly 80% of the queries because the queries include nothing but onomasticons and/or they are too short (under-articulated).

The Sequence Approach

The second innovation used in the CO is the use of sequences instead of single words. A single word, like “kill”, is the most ambiguous state of information and is hardly used in human communication without a strong underlying/implied context. As a result, building a natural language processing (NLP) systems by taking single word as the unit of computation is an invitation for disaster.

In contrast, word sequences (2 or more words) are inherently safe and highly descriptive. Take “road kill”, for example. This sequence describes a corpse of an animal killed on the road by a passing vehicle. If a language processing system takes the sequences as unit of computation, 99% of the ambiguity problem vanishes. There is no need to process the word “kill” and “road” separately, trace their senses, and locate convergence to identify the meaning of “road kill” if you can just take the sequence “road kill” itself as your unit of computation for mapping. This is depicted below:

Note the number of traces required in a conventional ontology approach compared to the sequence approach. The sequence approach requires a lot of data storage space (which is dirt cheap) whereas the conventional ontology approach requires a lot of CPU for a simple mapping task (which is expensive). But the bad news does not stop there. The trace routes in conventional ontology requires manual work (impossible to automate) whereas sequence-based ontology can be easily built via automation.

I realize only a handful of people will understand the second point above. Nevertheless, the scalability and performance of the end product will speak for itself when we put the testing platform on-line.

Usage of the Commercial Ontology

The immediate use of the CO is related to search queries, or document characterizations, that are not tied to any advertising in conventional systems. This unrecognized domain of search queries and characterizations means loss of revenue. hakia’s CO is designed to fill this gap. For example, if the search query or page characterization is “beat generation” the CO can map it to “literature” on the fly. As a result, systems using the CO will have much deeper understanding of the incoming terms, thus will be able to recognize the underlying intent beyond the face value of the words. The same capability can be used in a number of places other than advertising with the same effect.

Stay tuned for the release of the first version of our commercial ontology.

In the wake of SemTech09 conference, I thought this title would do justice to those mischievous readers who happened to have the good fortune to stumble across this blog posting. The conference was great, neatly organized, carefully secluded in San Jose, California. One of the highlights was the Semantic Search Keynote Panel with all the players on stage (Ask, Bing, Google, hakia, TrueKnowledge and Yahoo!) as seen in the picture below.

Bear in mind that semantic technology to “any” audience can be as heavy and stifling as what the topic of stem-cell research can be to the high-school students. Thanks to Carla Thompson from Guidewire who did a terrific job to come up with discussion topics and moderating the panel, everyone survived the ordeal without any sign of dozing.

Despite the positive outcome, some responses from the panelist made me wonder if we should go back to the basic question of “What is semantic search?” Or, better to discuss: what is NOT semantic search? Here is my list:

Structured data. Folks, structured data is NOT semantic technology. A database that can pull out a list of beer brands, their manufacturers, and their contact information, given the query “social drinking”, has nothing to do with semantics. I say this because some people seemed to be under the illusion that there must be some kind of semantic technology if a search engine brings such structured data in SERP. It is a trick as old as the ancient Egyptians who used beads on strings to organize harvesting information. Organized information is not semantics.

Morphology. If a search engine is robust (brings the same results) to a query “top ten” versus “top 10″ by recognizing “ten=10″ it would be a stretch of imagination to call it semantic. Anyone can come up with such a replacement list without a drop of linguistic knowledge. Similarly, distinguishing the name Fisher from the noun fisher by detecting the capitalization of the first letter does not go beyond the application of simple linguistic rules. These capabilities are not semantic search capabilities.

Syntax. It is true that certain level of semantic information can be salvaged from syntax. Unfortunately, if syntax was enough to detect the meaning of text, then an 8 year old kid who developed a perfect reading skill (syntactically parsing strings of letters and words in English) would be expected to understand the meaning of Shakespeare’s works. The difference between reading and understanding is the difference between syntax and semantics. Former requires the skill to parse things out, whereas the latter requires vast amount of associative knowledge.

Statistics. An infinite number of monkeys with a keyboard would eventually type the complete text of the declaration of independence. This is statistically correct. However, if a search engine is expected to become semantically apt using statistical algorithms, one has to wait until the monkeys finish their job. There is no place for statistics in semantics. For example, let’s take this sentence: “Polar bears don’t eat alligator eggs before dawn.” I am sure you have never seen this combination of words before in your life. But, the fact that you can understand what it means is simple evidence that semantic brain does not need statistical sampling. Meaning does not emerge from statistics. It emerges from associative knowledge.

Scalability. Scalability is the narrow bridge between science and technology. What you can carry from the science side to the technology side over this bridge determines the level of capabilities in real world. The science of semantics is huge stemming from the basics of philosophy. But, Web search is a highly particular problem with stringent constraints (narrow bridge). Designing semantic algorithms to drive a Web search engine is like walking on egg shells and requires a completely new approach. Therefore, a semantic algorithm can be very sophisticated but it does not mean it is a semantic search algorithm suitable for the Web.

The five issues I addressed above explain what is NOT semantic search and should guide the interested readers to question emerging technologies in SemTech10. Structured data, morphology, syntax, statistics, and scalability are the key questions to discuss. Obviously, no one would be afraid to ask these questions unlike what the title suggests, but if you understood the title, it was your semantic brain in action. That was my last example to “what is semantics” in this article.

Bing, the new search engine from Microsoft just went live and in doing so introduced a similar version of hakia’s categorized search. At its launch in 2006, hakia became the first search engine to provide categorized aspects of search queries via hakia Galleries.

hakia Galleries received industry accolades after their formal introduction in 2007. Our goal has always been to take search beyond 10 blue links. It was then no surprise when Microsoft invited us to show them the inner workings of the hakia Galleries in July 2008- shortly after their acquisition of Powerset. But it was a huge surprise to recently find out that Microsoft introduced categorized search in Bing. Today we checked out the Bing preview and compared the Bing’s categorized search feature to its inspiration, hakia Galleries.

hakia Galleries provide categorized aspects of search queries. For example, if you are searching for Obama, you can find information about his official site, headline news, images, biography, speeches, and more (see image below). Powered by semantic search, hakia Galleries prove 17 aspects of this query. We save the user time by answering 17 Obama related questions in one search. Compare the hakia Obama gallery with the same search at Bing.com (Bing provides only 7 aspects of this search query).

Let’s look at another example. Search for lung cancer at hakia and Bing. hakia provides the searcher links for the following aspects of this query: Basic Information and FAQ, Image Search, Headline News, Symptoms and Diagnostics, Treatment, Procedures, and Therapy, News, Clinical Trials, Healthcare Facilities and Finding a Physician, Alternative Therapy, For Kids, Research and Statistics, Organizations, Message Boards, and Images. Compare that with Bing’s aspects: articles, symptoms, treatment, prognosis, stages, clinical trials, and images. Look familiar?

As Danny Sullivan put it aptly in his Bing review, “Probably the most significant change is that Bing now organizes search results into categories (gives Obama example). The concept of grouping results also isn’t new. Long known as clustering, you can see it in operation at hakia (see Obama there) or Clusty (again, see Obama there).”

At hakia we could not dream of a marketing budget of $80-100 million. But hey, if you are out there to try Bing as an alternative search engine to Google, give the original categorized search a try at hakia.com (one of Bing’s inspirations!). You can surf the hakia Galleries here: http://gallery.hakia.com/ or try your search at hakia.com when you bing and ding.