Qweery's look at the world of search

Is an internal search engine the same as web self-service or can/must they co-exist? I believe they, if both implemented correctly, must co-exist. I’ll try and explain why.

First, what is my definition of internal search and web self-service.
An internal search engine works like Google and shows all pages which contain the search terms, ranked by a certain mechanism designed for the specific type of pages.
Web self-service is related to customer service and answers customer questions automatically without direct contact between the customer/visitor and the contact center of a company. With web self-service a customer can enter his/her question in normal sentences or a keyword like manner. Some kind of language processing is used for query formatting.

Both functions use technical concepts of information retrieval, spelling correction and synonyms, stemming and other word functions, which can make them a bit alike. An internal search engine can be used to act like a web self-service engine and the other way around, but I wouldn’t recommend this. Now my arguments why:

What happens when you use an interal search engine for web self-service? No language processing (NLP), limited control on which answer is given with which question, no content management but only crawling for content (possibly with a direct database connection). All typical search engine lacks for web self-service.

And what for a web self-service engine as internal search engine? Website page are not suited for answering questions only for additional or background information. Users expect a keyword search mechanism which lowers the language processing power for relevancy. Language processing lowers search speed for large collections. Manual control for large collections becomes a full-time day (a night) job.

Web self-service engines function best with 250-1000 answers to about 5000 questions users may ask within one organisation.
Internal search engines function best with >250 collections of webpages which can be similar in content.

Web self-service engines should be presented in a separate customer service environment on a website and only present specific answer to specific questions. If there is not answer present, it could redirect to a internal search engine, but only if the users tells it to do so. This increases the reporting power of the self-service tool immensly. The self-service tooling can/must also be used at any webform used on the website, like checkout pages to enable the user to ask proces related questions without leaving the proces it self.

The internal search engine should not index the self-service answers and suggest the use of the tool at the customer service pages and a search box should be placed on all webpages at the top right corner. Use the search engine for product search for instance, increases the controlling power of the administrator and with the right search engine also the conversion for the products in a webshop.

Correctly implemented internal search and web self-service complete each other and make the website stronger and more usable for both visitor as administrator.

Even before Google, Altavista used the search channels ‘Web’, ‘Images’ and ‘News’ to let users devide search results. Google adapted this approach and I cannot understand why this remains.

I believe about 80% of all searches is done on the Web-channel and the search results become worse and worse. My main focus of search privatly is in Dutch and we cheeseheads seem to be into the secondhand markets….a lot. So what happens: anything you search for results in the first 10 pages in Google are ads from these market places (marktplaats, tweedehands, etc). Very frustrating if you are looking for productinfo from the original supplier.

My suggestion is to alter the channels into something like: general web, discussions, markets. This could be worked out in more detail but you get my drift. We incorporated this in the public Qweery search engine 3 years ago and got a lot of positive feedback on it. Not to tap ourselves on the back, the public engine won’t be here to see 2009, but I think this will be a true addtion to the mainstream search engines like Google of LiveSearch.

Now to think about how to determine which page goes where.
The way we did this at Qweery was: markets were identified by domain, so we kept a large list of domains which we believe are marketsplaces. A lot of work, but does work well.
Secondly the discussions or better: fora and blogs. Wel, this can be done much easier: just look at the url and if it contains forum or blog then put it in the channel Discussions. Maybe not very clean, but it seems to work.
I can image sites to abuse these kind of rules, so some other great thinkers should work this out in more detail, I guess.

What do you think?

Cheers,
Maarten Rooseboom