When it comes to searching across the web, we all know that Google is king, but does this still hold true across your own internal network?

Over the past 12 months we have wrestled with this question, particularly in an environment with multiple search mechanisms, manually maintained indexes, and masses of sites that were created when metadata was primarily used to categorise instead of search.

In a series of posts, I’m going to go through our experiences in improving search across our internal network – I’m not suggesting we have found the magic search bullet, or that we’re anywhere near finished tweaking and tinkering, but I do know we’re a hell of a lot closer than where we were at this time last year.

The problem

In our travels across campus, we kept hearing “I can’t find what I’m looking for!”  – not surprising, given that we had;

1. 500+ individual sites, ranging in age (earliest was 1997), metadata (none to Dublin Core to ‘something’) and ownership
2. Inconsistent use of key SEO elements, such as title, headings, tags and meta descriptions across the majority of our sites
3. Multiple sources of content and internal search mechanisms, each with their own set of search results
4. Manually maintained indexes, all categorised and sub-related, together with an in-house redirect mechanism
5. An internal audience of staff and students with heavily cyclical search requests – a search for ‘physics’ at the beginning of semester is more likely to be for text books, and at the end of semester past exam papers

Image: Multiple source-centric result sets;

Given Google’s dominance in search, we quickly went down the path of a Google Search Appliance, or ‘Mini’, which is a self contained rack mounted system that gives you God-like powers over the Google algorithm. We were bringing in a little bit of Google in to magically transform our disparate set of sites into a cohesive set of search results.

Once plugged in, the Mini worked really well – for pages that were properly formatted for organic search.

Pages that were missing or incorrectly using titles, headings and metadata didn’t fare so well, and we found the search results were not the most relevant, as the Mini couldn’t make much sense of most of the content it crawled. We also found that there was no clear way to incorporate the feeds from other systems, with the “how do I…” answers primarily provided by a community of Search Appliance users and resellers, and not Google themselves.

Given the wide ownership of the sites we were working with, updating each with appropriate SEO friendly content was unrealistic. What we needed was a way to;

1. compensate for the lack of SEO content,
2. incorporating multiple sources/ formats of content,
3. allow for cyclical requests to ensure the most relevant results appear, and
4. combine all the different sources of search results into a single set of user-centric search results.

Enter Adobe Search&Promote

If you’re a regular visitor to this blog, it will come as no surprise that Tim is a power user of Omniture products, steadily working his way around the product wheel. We became aware of the Search&Promote product (then called SiteSearch) which promised to solve our key internal search issues.

Search&Promote uses a search algorithm to organically crawl your sites, in addition to ranking rules based on a wide range of configurable data. Once you’ve defined your rules, you can adjust the overall balance between your ranking rules and natural search relevance.

Where there is a lack of metadata, Search&Promote can be configured to dynamically inject metadata on crawl, based on a URL pattern. Additional custom metadata can also be injected to create facets (filters) that allow users to drill further down into predefined categories.

If your multiple sources of content can be transformed into XML feeds, then that content can be crawled, categorised, and integrated with the organic results by Search&Promote.

Yes, there are other internal search products on the market that will do the above, however there is one thing that Search&Promote has over its competitors – the ability to tightly integrate with SiteCatalyst and Test & Target.

We’ve known for some time that internal search terms follow highly cyclical patterns as our student (and staff) needs change over the semester. We’ve helped them find what they’re looking for using of real-time SiteCatalyst data in search-as-you-type and tag cloud mechanisms, however with Search&Promote we now have the opportunity to take internal search to the next level.

In the report below (7 day moving average) you can see two popular search results across three semesters peaking at different times during the semester;

Notice how ‘bookshop’ peaks at the beginning of semester, then dies down, only to peak again at the beginning of the following semester. No surprises here, but it does coincide with a significant increase in page views across the Bookshop website.

Then look at the results for ‘timetable’ – there’s a peak at both the beginning and end of semester. The difference here is that people are actually looking for two different pieces of content – their semester timetable at the beginning, and their exam timetable at the end – using the same keyword. Again, the rise in search terms coincides with increased page views across each piece of content.

So, in theory, by looking at the last week’s worth of traffic across our group of sites, we should be able to determine what content students are looking for, then re-rank the search results accordingly. For example, the term ‘timetable’ at the beginning of semester will push results related to the semester timetable to the top, and at the end of the semester push results related to the exam timetable to the top.

Exciting stuff!

Next post – the implementation.

Search is a veritable gold mine that is frequently ignored.

I’m not talking about Search Engines and Keywords, I’m talking about your internal search. Providing you track internal keyword searches, you can gain a wealth of understanding.

Internal search is generally used as a quick wayfinding method, highlighting areas of content that are well used, but are not readily available.  And more often than not, it’s seasonal as well.

The following report shows the top 5 keywords used by students on our University website, smoothed using a 21 day moving average.

Two things immediately jump out.  Firstly, notice how searches for Bookshop increased dramatically in late February, then declined, then Library picked up substantially in March, followed by Exam Timetable in late April (spiking suddenly in late May).

Very seasonal activity, centered around the Student’s life cycle at the University.  In February, having started a new semester, they all needed books.  Then as semester progressed, they all needed access to the Library, and lastly, as exams approached, they all wanted to know their Exam Timetables.

Great – but how can we use this?

Well, firstly this points to the fact that there is no clear navigation on our site to these destinations.  There’s actually a reason for that – our public site is not designed for active current students as they should be using the Student Portal.  That aside, we can assist though in making these destinations easier to get to.

One thing we can do with Omniture is to export these top keywords to our database on a daily basis and then represent them on the site, through things like tag clouds and quick search results, making their lives easier in the process.

By using real data and automating the process, the seasonality of searches will also come through to the site as well.

The other interesting thing about the above chart is their search for timetable, as opposed to exam timetable.  When they start a new semester, they are interested in their class timetables and frequently search for them.  As time progresses and the memory kicks in, that activity slows down.  Again, this can be used in a number of ways – not only online, but through mobile apps and offline support media as well.

In order to track search keywords, simply put the keyword into an s.prop.  You might also want to put the number of results into another s.prop so that you can cross-reference the results on a keyword by keyword basis.  This would highlight any search terms  that return no results.

Segment your keywords

As always, understanding your audience is critical online.  If you segment your audience types, you’ll also be able to see the different searches conducted by different audiences.  We do this and the results are (obviously) very different.  But, without the ability to do it and show it, you’ll be taking an educated guess that they are different.

To do this, put the audience type into another s.prop and use the getAndPersist() plugin to set a cookie.  That way, all activity can actually be segmented by audience type.

Now you can provide some really smart assistance

When you start to build your tag clouds or your quick search lists, you can base the results on the value in the cookie, thereby customizing the information to the user and making it more relevant to them.  Just a few little tricks like this and you’ve helped to increase the usability of your site, which will go along way in driving customer loyalty (and hopefully revenue).