Qweery's look at the world of search

If you run a webshop your goal is to sell a much of your products as possible. So you create a great looking website, add your products and focus on getting found in Google (and/or use adwords) to get visitors. You have optimised your website so conversion is high: your products are easy to find, well documented and visible (images/video) and your checkout process is easy. So where is your datafeed? What? Yes, your datafeed…..PUBLISH YOUR FEED!

What is a datafeed?
A datafeed, also known as product feed, is a file containing all information of all the products you sell. The format of this file is often XML and can look something like this:

The feed should contain as much information as possible related to the products you sell. It can be something like a RSS feed but be aware that the productfeed I am talking about MUST contain all productdata for all products you have. If the file becomes too big, split it into several files (make a sensable split by producttype or something).

Why publish your datafeed?
Well…to sell more! How? There are a lot of websites out there, who are willing to add your productfeed to publish on their own website…for free! True, a lot of websites only accept affiliate productfeeds to earn some money with a datafeed, but to complete the offering on a vertical (websites where all sites for one subject are indexed) your datafeed will be pickedup.
And if you are thinking about using affiliate someday, your feed is ready. By letting others use your content, you will sell more…and that is the goal.

How to publish your datafeed?
Just put it on your website. Add it to the menu or the links at the bottom of your pages. Just make sure it can be found easily…some nice bonus: search engines will pick it up as well.
Most webshop software can create your datafeed automatically from the database it holds. Just make sure the feed is complete, correct and up to date and you are good to go….and sell more!

A lot of website use forms of attribute search to help their users find the information faster. Examples are categories, price ranges, supplies, brands etc.
Formerly, the most common way to let searchers use these attributes was with drop-down selectboxes. The last year you see the faceted search type more and more. I believe the power lies in multi faceted search. What’s that? Let me explain:

Faceted search works kinda like this: a users types in a searchquery and the website shows the result engine result page (SERP). Next to this resultset, the engine shows all possible attribute values (grouped) with the count of available results shown in brackets:

By clicking on one of the atrtibute values, the searcher quickly narrows its search….quickly is the general idea…however, it can result in a endless clicking exercise. Let say you are looking for clothing for your baby and the webshop search offers faceted search. You type in ’shirt’ and it shows you about 5.000 results. The atribute filter contain size. Thank god, this should speed up things. What size does she have now…hmmm..62, 68…Let me try 62 first. The resultset is filtered and no options for sizes are shown anymore. 10 pages and 6 filter options later: nothing. Ok. Let’s click back to my original ’shirt’ query and try size 68….you get the drift.

Another way, not faceted, is to let users select more than one value of a single filter with multiple select or checkboxes to be able to select size 62 AND 68 together. I don’t come across this way of searching much, but I think it is a lot better than being able to select a single value. Main issue with this kind of search (which I have seen) is that they don’t show the number of expected result by selecting one of more values.

We are now working on multiple faceted search as a standard option within the QweeryBox. We already offer multiple selection and faceted search a separate filter options but in our next release we will have combined these two and enable an even better search experience. When will you see this in public? At the end of july we expect to launch the new version of boezoe.nl, bases upon the latest version of the QweeryBox and it will have multiple faceted search!

Happy searching,

Maarten Rooseboom

A lot of blogposts start with a question to use one or another method to improve search quality. This post is like that: a lot of search engine (and alike) suppliers of corporate engines offer the feature of automated synonym usage to expand user search query’s. This seems very usefull since the contentsource doesn’t have to be expanded and a user still finds what he’s looking for…..is that true?
I believe not.

The dutch vocabulary consists of roughly 200.000 words. Offical synonym files contain up to synonyms for up to 80.000 words. These synonyms belong to the official dutch standard and that is exactly where the problem is. Since the engine is used in a corporate environment (internally or on a website) the context of content is limited while to synonym list is not.

A search query standard will be expanded with all known synonyms of every word in the query. This means all synonyms can trigger a search result and end up in surprising, unexpected and more importantly bad results. And because these list a so big it is unmanagable for a webmaster (or search optimiser).

Why not turn it around: in most cases about 250 terms in a single corporation are up for expand words, like synonyms. And in most cases these terms are so specific (productnames, branche related) that the language standard doesn’t have the correct synonym.
So how much work is it to manually add these terms to a vocabulary of the search product? Not that much…a single day probably is more than enough….think about it.

Updating our public label BoeZoe to crawl and index all childrens clothing in the Netherlands is quite a job.
We are working on the second version of the website and the biggest adjustment is the possibility to choose more than one item per filter.
For example sizes: most women (sorry guys, women mainly use this website) look for sizes like 50-56 AND 62-68. Because those little ones grow so fast and not all clothing are the same size (even within the same category) they want to search for more than one size at a time.
The second big change was the looks, for a sneak preview I’ll show both designs (old one first, new one second) below.

Current (old) version 1

New version 2

The new version is not live yet, we are testing it right now. And this is what we run into with our new filters: choose a filter and it won’t search, no it will select all items with that meta data. So it won’t do a search in the content for example the brand. This gives an unexpected result…well it did at first, but it is quite logical: there are 100+ shops we index and the use of a filter showed the clothes sorted by shop instead of best suited for the search query (which is not there wheen you don’t use the searchbox). Hmmmm…this is not what we want.

Should we now change our filters into predefined search words? I think we do. We’re gonna test it anyway. This will (as we expect) result in the regular search query and our algoritm will sort on best match in content.
We won’t do all filters this way. Brand, color, clothing type, size will be predefined search words. So if you select size 50-56 it will search for (maat 50 | maat 56 | mt. 50 | mt. 56) and the manually typed query will be added to that.
For pricing, state (new/used) we will keep the meta filters. Gender is still undecided at this time. We will test both and choose then.

So, if you want to build your own vertical search site, remember the use of meta filters is not always the best way to let user experience your search platform. It is a bit more work to manage predefined search words but the results are more as users expect.

Cheers,
Maarten Rooseboom

Plain websearch on public engines like Google, Yahoo and such or on a website mainly consist of a single input type text box to type a search query. In some cases this is extended with a category filter and the use of advanced search. Is this user friendly and smart to do or should you incorporate something else?

Well, I think a plain search is not the smart thing to do. Several researches show that about 50% of visitors on a website use the searchbox to find the information they need. So why let them swim with simple terms and not guide them and give them the search scope they need. For instance a shop for pda’s like www.pdashop.nl . The hold several brands in theire catalog but on the starting page for search only show a tiny searchbox. I would advise to put some filters right there with the searchbox. You can think of several filters to put up there:  brand, weight, pricing, engine and some additional attributes pda’s tend to have. These filters should operate on themselves as well, so without a search query in the main searchbox. THis gives the user some additional browse capabilities. Since selling pda’s (in this case) is about everything this shop wants to do, it should focus on the shortest way to get the potential client to the preffered pda.

In the case of public engines Google holds a single searchbox and claim to be able to understand what a user means with that single query and know what context the user holds. This is impossible and as I stated in an earlier post, the use of channels would be a start (information pages, discussion/blogs and forums and shop/auction pages). Google furthermore offers extended search in which you can tell the engine to search in a single website, look for specific file types and such filters. These a quite useless. Most of the time a user doesn’t know which website the information can be found, doesn’t care which  filetype. These filters a typically thought of by techies and should be reconsidered in more usefull terms, more categories, pricing filters (for the shop channel), look for discussion with more than one message in a singel topic. Companies like Google and Microsoft should be able to incorporate this technically and have the resources to do so.

A single searchbox would only be usefull for a site up to 50 pages with nothing but textual information otherwise the immediate use of filters would make a search much faster and more  succesfull.

The discussion on wether or not to automatically alter the users search query to one the search engine thinks is correct is subject of a new discussion now that Google seems to be doing this with their “Did you mean” function.

What happens? If you type a query for which Google finds a suggestion, it now shows 2 results for the suggested query at the top of the search results. Below you’ll find the results for the query entered by the user.

Automated correction results by Google

How thoughtfull? I think not, not for a public search engine anyway.

What makes the function ‘Did you mean…’ tick? Well, most probable both present worddata found in all documents indexed by Google and user input combined. Every query is checked with known words and a most common query is build from popular searches similar to the one checked and with more possible results.
Since Google has no limited information domain (it indexes all info found in the world) a particular search, like the one shown in the image above, cannot easily be translated to the one suggested. The subjects are totaly different, only the letter combination is one letter off.

When is it possible to do an automated replacement? Within a website for instance. The information domain on a website is limited and a suggestion comes from information of that website (so don’t use the common API from Google ) and would probably be sensable.

Preferably the user is given an option to save his or hers preffered setting for a next time. An advise to Google: add this as an option to the user preferences!

Is an internal search engine the same as web self-service or can/must they co-exist? I believe they, if both implemented correctly, must co-exist. I’ll try and explain why.

First, what is my definition of internal search and web self-service.
An internal search engine works like Google and shows all pages which contain the search terms, ranked by a certain mechanism designed for the specific type of pages.
Web self-service is related to customer service and answers customer questions automatically without direct contact between the customer/visitor and the contact center of a company. With web self-service a customer can enter his/her question in normal sentences or a keyword like manner. Some kind of language processing is used for query formatting.

Both functions use technical concepts of information retrieval, spelling correction and synonyms, stemming and other word functions, which can make them a bit alike. An internal search engine can be used to act like a web self-service engine and the other way around, but I wouldn’t recommend this. Now my arguments why:

What happens when you use an interal search engine for web self-service? No language processing (NLP), limited control on which answer is given with which question, no content management but only crawling for content (possibly with a direct database connection). All typical search engine lacks for web self-service.

And what for a web self-service engine as internal search engine? Website page are not suited for answering questions only for additional or background information. Users expect a keyword search mechanism which lowers the language processing power for relevancy. Language processing lowers search speed for large collections. Manual control for large collections becomes a full-time day (a night) job.

Web self-service engines function best with 250-1000 answers to about 5000 questions users may ask within one organisation.
Internal search engines function best with >250 collections of webpages which can be similar in content.

Web self-service engines should be presented in a separate customer service environment on a website and only present specific answer to specific questions. If there is not answer present, it could redirect to a internal search engine, but only if the users tells it to do so. This increases the reporting power of the self-service tool immensly. The self-service tooling can/must also be used at any webform used on the website, like checkout pages to enable the user to ask proces related questions without leaving the proces it self.

The internal search engine should not index the self-service answers and suggest the use of the tool at the customer service pages and a search box should be placed on all webpages at the top right corner. Use the search engine for product search for instance, increases the controlling power of the administrator and with the right search engine also the conversion for the products in a webshop.

Correctly implemented internal search and web self-service complete each other and make the website stronger and more usable for both visitor as administrator.

Just try a search in Google for something on a search engine (perhaps with the addition of optimising) and SEO (Search Engine Optimisation) is all you get as results…it is very hard to find anything about optimising your own search engine although it is one of the most important function of a website….on article I like was (in dutch):
http://www.webanalisten.nl/analyse/bezoekers/investeren-in-zoeken-zinvol.html.

My guess is that SEO mainly only focus on getting higher ranking in Google and do this by optimising the content of the website, the link structure and most important: incoming links (this is not a SEO article so I’ll leave other more detailled options).

But what happens when you got that visitor, that lead from Google? Well…just picture what you do yourself. You judge the page you see and match it on the keywords you used in Google. If that is ok, you probably didn’t reach the exact page you are looking for (you don’t even know the site and have no idea wether this site contains the page/product you are looking for)….so what do you do? Two options: scan the naviation of hit the search function.
The search function is used the most (studies mention 50%+ )…mainly because the navigation just need to use the same words/terms you expect and almost never do.

So we’ve reached the internal search engine. In most cases a fulltext search function on the underlying database. Is this sufficient? For a very small website: yes….small meaning <50 pages in total. Is the site any bigger..no way. What about the ’special’ search options in webshops? Well, in most cases you get a category filter and that’s it. Still, search for any term and you’ll get hundreds of results….want to browse through all of them? Don’t think so and you’re off to another website.

What is the webshop manager to do? Well, check what you have looked for, check what result you saw and manage the results. Add synonyms, alter search terms for terms the shop uses, ignore some terms…is that enough? Uhm….nope. What about what other visitors did? Look at the big picture….Amazon was one of the first to use statistical information and showed visitors what other bought. You can use these ideas in search as well. Look for a search engine which helps you with this.

So, some pointers:
1) check what searches your visitors do (not in Google, but in you own engine) and make sure the content matches this
2) manage the keywords your visitors use and alter the query if needed
3) split product content with text pages in search with a searchfilter of in your resultset
4) make sure your best converting products top your resultsets
5) let important pages/products rank higher
6) show high marging or interesting promotions at the top of the results (not all, but keyword based).

Good luck…need any help. Let me know and I’ll have look if you want.

Even before Google, Altavista used the search channels ‘Web’, ‘Images’ and ‘News’ to let users devide search results. Google adapted this approach and I cannot understand why this remains.

I believe about 80% of all searches is done on the Web-channel and the search results become worse and worse. My main focus of search privatly is in Dutch and we cheeseheads seem to be into the secondhand markets….a lot. So what happens: anything you search for results in the first 10 pages in Google are ads from these market places (marktplaats, tweedehands, etc). Very frustrating if you are looking for productinfo from the original supplier.

My suggestion is to alter the channels into something like: general web, discussions, markets. This could be worked out in more detail but you get my drift. We incorporated this in the public Qweery search engine 3 years ago and got a lot of positive feedback on it. Not to tap ourselves on the back, the public engine won’t be here to see 2009, but I think this will be a true addtion to the mainstream search engines like Google of LiveSearch.

Now to think about how to determine which page goes where.
The way we did this at Qweery was: markets were identified by domain, so we kept a large list of domains which we believe are marketsplaces. A lot of work, but does work well.
Secondly the discussions or better: fora and blogs. Wel, this can be done much easier: just look at the url and if it contains forum or blog then put it in the channel Discussions. Maybe not very clean, but it seems to work.
I can image sites to abuse these kind of rules, so some other great thinkers should work this out in more detail, I guess.

What do you think?

Cheers,
Maarten Rooseboom

As of today, we at Qweery will post articles about search related topics. Some will be commenting on news, other articles will be to engage in a discussion with you. Keep track!

See you soon.
Qweery