So I promised that I would finally put fingertip to keyboard and talk a little bit more about using Visitor Scoring…to finish up the series that I started a while ago.

If you’ve read my previous posts, you’ll know that we implemented a series of metrics for engagement measurement, culminating in a per-visitor score.

I wanted to share with you some of the insights and benefits of doing all of this, particularly in Discover.

An Engaged Segment

If you remember from the previous post, we talked about using a total of seven different metrics to try to ascertain engagement, or, when used in combination, could identify an engaged user segment.

The final metric was a measure of interaction and there were two ways to implement; either by just counting the number of visitors that participated in specific actions on your site, or the second way; by creating a scoring methodology at the visitor level and leveraging that as the final metric.

A little tougher to do, as it involved some additional code across the site, but definitely implementable, and now, we’re able to use that information to gain even greater insights.

The reason you should segment is because not everyone on your site converts – in fact, probably only around 3-5% actually do what you want them to do.  So you use segmentation to analyse those that have converted and try to understand what made them different, what their different behaviours were, and if possible, try to use that as predictors of future behaviour by the other 95-97%, so that you can lift conversions.

Creating the engaged segment

Using Discover we created a Visitor container that contained the following rules, which we had previously determined worked best for our business.

The rules applied are from 6 of the original 7 indexes that defined engagement for us:

1. Click Depth Index – which captures the contribution of page and event views
2. Duration Index – capturing the contribution of time spent on site
3. Recency Index – which captures the visitor’s “visit velocity”—the rate at which visitors return to the web site over time
4. Loyalty Index- the level of long-term interaction the visitor has with the brand, site, or product(s)
5. Brand Index – the apparent awareness of the visitor of the brand, site, or product(s)
6. Interaction Index – visitor interaction with content or functionality designed to increase level of Attention

Notice the counts at the bottom of the segment creator – those seem to be a bit confusing as the number on the left (216 Visitors) represents the number of visitors that meet the criteria of the segment, but the number on the right (1,071,458 visitors) does not appear to be a count of visitors for the timeframe (for some reason I have yet to figure out).

Additionally, for this particular segmentation example, I wanted to ensure that our engaged visitors have actually become a lead (one of our primary goals).  So our segment includes that, as well as them having a score of greater than 200, which means they’re interacting with a fair amount of content or doing a fair amount of activity (such as applying to come to study with us).

Saving the segment, it’s then ready for use.

Using the segment

One of the first things I looked at was where our engaged users were coming from, their average scores, conversion rates and application rates (or purchase rates), compared to all visitors who are not staff or students.

And I fell off my chair.

While the number is quite small comparatively, due in part to the timeframe I’m using, the score, lead conversion rate (leads/visitors) and application rate (applications/visitors), are significantly higher.  This is to be expected due to the segment, but I wasn’t quite expecting the numbers to be that high.

Visual Site Analysis

I was keen to understand, visually, where these highly engaged users go on our site, so I used the Pathing Site Analysis report in Discover (one of my personal favourites).

Firstly I started with all visits to get a sense of what they do across common pages that we want them to interact with:

Fairly widespread usage of key pages.  The thickness of the arrow indicated volume of traffic from one page to another – the bulk of traffic goes to the Courses homepage, from the site home page.

Next, I applied the same engaged segment:

Ok, I must need velcro pants because I fell off my chair again.

It appears that our highly engaged users take a whole different path.  The colour indicates their propensity to become a lead.  The big red steps are basically the lead capture process through our main tool – Figure Out Your Course.

And they seem to browse around first within courses then become a lead – which is also good to know.  But once they become a lead, they tend to leave the site – which means we need to ensure that we’re effectively communicating with them through other channels, such as email at a later date.

Breaking down the traffic sources

In breaking down the Organic Search by time spent, we also see that a significant portion of highly engaged visitors spend more than 10 minutes on the site, and are more likely to convert to leads having done so, as compared to all visitors – another significant insight for us to use.

In summary

This is just one example of many that could be used on your site.  Once you’ve identified your engaged users, you can segment them further by demographics, or by customer type, or by content viewed, or by member/non-member and so forth.

You can view revenue by engaged/non engaged (bound to be vastly different), or average order value etc. 

If you use Test&Target, you’ll be in a great position to leverage the engaged user segment, targeting non-engaged users differently to increase their engagement levels.

All of this will help you gain better understanding into their behaviours, so that you can then further optimise your site to improve those conversions.

I’m keen to hear from others that have used Visitor Scoring, or Engagement Metrics across their site, coupled with Test & Target to lift conversions.  Let me know what your thoughts or successes are.

This is the third (but not final) post in the series on Visitor Engagement.  One of the problems with the Visitor Scoring method that I previously described, is that, at the end of the day, you’re still somewhat limited to viewing scores at the “average” level, by segment.

That presents a number of challenges because the average is precisely that…and the underlying scores vary dramatically within each segment.

But there is a way to see what each and every visitor score is, or even within the different segments…and it’s called the Unique Visitor ID.  You can see at the visitor level, how many times they’ve returned, how many “things” they’ve done, such as searches, product views, revenue etc.

This is really the icing on the proverbial cake.

Visitor ID

There’s a seldom used implementation method in SiteCatalyst called Dynamic Variables which easily enables all of this capability.

All you need is a spare eVar and you’re in heaven.

Step 1

Simply populate an eVar with “D=s_vi” and the result is that SiteCatalyst will populate the eVar with the unique visitor cookie ID.  If you test it with WASP or something similar, you’ll only see that D=s_vi got passed, but the magic begins.

s.eVar56="D=s_vi";

Step 2

Go into you Admin console, and on the report suite where you want this collected, go to Conversion > Unique Visitor Variable section.

You’ll be presented with a dropdown of available eVars that can be selected to use as the Visitor ID…simply select the one that you’re using and hit save.

That’s it – you’re done!  All there is to it.

Rest assured, it does not impact anything to do with the way that SiteCatalyst measures unique visitors – its completely separate and safe to use.

And now it pops

So, the first thing you’ll see is a new report on your eVar. The report is a list of all Visitor Cookie ID’s captured during the timeframe selected.  As it’s a conversion variable, you can add in whatever success events you like:

Add in your Visitor Score and you can see what their points are.  As you can see from above, I’ve highlighted an interesting visitor (#7)…they were on their 2nd visit, they searched 12 times, viewed 16 courses and submitted an application.  Their score is currently 542.  I’m wondering if visitor #5 is a candidate for an app too…similar behaviour is forming.

Segment-based Visitor ID’s

Ok, now open up a report such as Campaigns (we use the Unified Traffic Sources VISTA rule so it may look a little different if you don’t use it).

Remember the challenge I mentioned at the top of this post…with average scores.

We see below that the average visitor score for Organic Search is 7.88.  But, that’s just an average.  We know that from our scoring methodology, you have to have a higher score than that if you’re interacting with content.

So, when we break down the Search Organic by Visitor ID’s, we see a very different story.

Look at that…the top 10 all have scores above 300!

So to get to an average of 7.88, we must have an awful lot of really low end scores that come in through organic search.  And actually, that makes sense…we have a lot of reference content across our site, such as Library and School content that is very popular and that’s likely where these users with low scores are going.  Not the ones we’re really interested in from this scoring standpoint though.

Extracting Visitor ID’s

The next fancy bit of functionality, once you’ve enabled the Unique Visitor ID, is the ability to extract those visitors that have met a certain criteria (success event).

If you now click on an item in, for example, the Last Campaign report (at the individual campaign tracking code level), instead of heading off to the summary page for that item (which by the way, is useful also), you’ll get a new menu:

If you select the success event you want the visitor ID’s for, you’ll end up in a data warehouse data extract screen, where the report can be scheduled for immediate delivery:

Here we’re extracting visitor ID’s for those that submitted an application having come to us from Google Australia.

The Icing

I think the real icing on this cake is the ability to look at all visitor scores at the individual visitor level, irrespective of the segment, so that you can get an idea of what their scores are, to help better gauge those low/medium/high engagement buckets.  It’s also really useful to see at the segment too.

You’ll need to use Excel to process the data (and one thing I should point out is that an eVar can only have a maximum of 500,000 unique values in a given month – so if you get more visitors than that, then this might not work for you).

Other ways to use this is to extract emails that have responded to a campaign – if you capture the memberID in the eVar you can then extract the data for later use.

The next thing we need to do is re-configure our Discover segment for the interaction index and amalgamate these metrics into the overall engagement scoring methodology.

Now, there’s a hot topic.  Measuring engagement.  One of the most widely debated topics in web analytics.

What is engagement and how do we measure it?

Engagement, unfortunately, is not derived from a single measure.  It’s not time on site.  It’s not how many pages they viewed.  It’s not bounce rates and it’s not about conversions.

Engagement is about a lot of things.  What is an engaged visitor and how do you measure engagement?

“Visitor Engagement is an estimate of the depth of visitor interaction against a clearly defined set of goals.” Eric T. Peterson and Joseph Carrabis.

A while ago, I came across their paper through Web Analytics Demystified, entitled “Measuring the Immeasurable: Visitor Engagement”.  While I won’t go into it in any detail, I will suggest that you read it, as it’s the background of this post.

The premise of the paper is that visitor engagement is made up of 7 different metrics, and expressed through one formula:

where:

Engagement can be expressed as the average of the sum of indexes, across specific segments, according to a:

1. Click Depth Index – which captures the contribution of page and event views
2. Duration Index – capturing the contribution of time spent on site
3. Recency Index – which captures the visitor’s “visit velocity”—the rate at which visitors return to the web site over time
4. Loyalty Index- the level of long-term interaction the visitor has with the brand, site, or product(s)
5. Brand Index – the apparent awareness of the visitor of the brand, site, or product(s)
6. Feedback Index – qualitative information including propensity to solicit additional information or supply
7. Interaction Index – visitor interaction with content or functionality designed to increase level of Attention

According to Eric T. Peterson, “Visitor Engagement is a function of the number of clicks (Ci), the visit duration (Di), the rate at which the visitor returns to the site over time (Ri), their overall loyalty to the site (Li), their measured awareness of the brand (Bi), their willingness to directly contribute feedback (Fi) and the likelihood that they will engage in specific activities on the site designed to increase awareness and create a lasting impression (Ii).”

When applied at the visitor level, on a per-visitor basis, they combine to form a pretty good proxy for visitor engagement.

Having read the paper, I was intrigued, and decided to use Discover to implement this…to some pretty insightful results.

A summary of the indexes

Click Depth Index

The percentage of your overall audience that has a minimum threshold of an acceptable number of page views per session.  If you see that on average, visitors “convert” after viewing at least 5 pages, then your minimum threshold would be 5 pages per visit.

Duration Index

The percentage of your overall audience that has a minimum threshold of an acceptable amount of time on site per session.  If you see that on average, visitors “convert” after spending at least 10 minutes on your site, then your minimum threshold would be 10 minutes.

Recency Index

The percentage of your overall audience that returns and converts within an acceptable amount of time (generally days).  If you notice that most visitors convert between 1 and 10 days, then you’d be looking for visitors with a return frequency of <= 10 days.

Loyalty Index

The percentage of your overall audience that has a repeat visit frequency in excess of a minimum threshold.  For example, if you notice that many visitors convert after visiting your site more than three times, then your threshold would be a visit count of at least 3.

Brand Index

The percentage of your overall audience that comes to your site either directly, or through branded search terms.

Feedback Index

The percentage of your overall audience that completes feedback on your site, or participates in rating or reviewing content, or commenting on blogs.

Interaction Index

The percentage of your overall audience that interacts with specific content on your site, or engages in an activity on your site.  There are no thresholds for this index – they are simply counts of.

Note: while you can count pre-defined activities on your site, it is better to score visitor interaction.  I’ll be doing a post on visitor scoring shortly.

Using Discover

Discover was built for this!  It’s very easy to create segments within Discover and apply them across various views to gain insight.

Firstly what we did was to look at some of the thresholds to understand what our “Anonymous” segment of traffic does.  Our anonymous segment is made of non-student and non-staff traffic, which we already have segments for in both Discover and SiteCatalyst.

We figured out what our minimum page views per session should be, the average duration, frequency of visit etc, by looking at them from a conversion standpoint…i.e. how many pages does a converter see, on average, before converting.

Once we’d done that, our 7 segments were easy to define as follows:

1. Click Depth Index – Visitor container, Path length > 10
2. Duration Index – Visitor container, Seconds spent per visit > 1800 (30 minutes)
3. Recency Index – Visitor container, Return Frequency <= 7-14 days
4. Loyalty Index – Visitor container, Visit number >= 2
5. Brand Index – Visitor container, Organic Search Keyword contains “Murdoch” or Visit without referrer
6. Feedback Index – (we don’t use this one)
7. Interaction Index – Visitor container, any of the following events: Lead Complete, Application Complete, Form Complete, Tool Name

The 8th segment was All Visits.  In each case, we used the Visitors metric to view the number of visitors that were part of each index.

If we view this against referring sites, what we end up with is the number of visitors that match each segment rule:

Export to Excel

What we need to do now is export the data to Excel to do the averages and generate the final engagement value.

Simply select the first item “None”, click Ctrl+A for select all, then click Ctrl+C for copy.

Open Excel, and paste the raw data into a new sheet.

Then it’s simply a matter of calculating one columns percentage as a percentage of the All Visits – Visitors column.

Once you’ve done that for each column, you have the indexes for each segment.  Now you just average all of the indexes to get an engagement metric.

For example, we see in the above that Direct Traffic, “none” in the above report, has an overall engagement value of 23%.  But if we look at the other columns, we also see that they are at the median value on Click Depth, whereas traffic from deewr.gov.au is well above the median.

A couple of interesting things have also been highlighted in the above, for example, traffic from Google Singapore is actually far more engaged than traffic from Google Australia – now that’s interesting.

Of course, you should always look at engagement via larger segments, for example, by Campaign, by Site, by Time of Day, Day of Week etc.

While Tuesday comes out overall for a better engaged visitor, I’ve highlighted other interesting things, such as on Saturday and Sunday visitors click more, but more visitors spend time on Saturdays.  During the week is better for branded search term visitors, and Wednesdays seems to be better overall for key interactions.

Multiple Sites

If you have multiple sites, such as microsite etc, you might want to check engagement across them to see if they are dramatically different, so you can then begin to try to understand why.

In the above calculation, I’ve removed the Feedback and Interaction indexes from the calculation, as they would skew the results.  It’s interesting that while the main University site has an engagement index of 23.67, versus a median of 20.43%, sites like Mobile and Maps have a very high engagement value.

Segmentation

Once you’ve got the basic engagement working and you’re looking at things from the overall perspective, you can then easily begin to look at engagement by different segments.

In Discover, you can create segments on-the-fly and apply them across your other segments, by simply dragging the new segment onto the filtered workspace. For example, we segment the above by Anonymous visitors, after we’ve built the overall segments.  We can do the same for Converting Visitors, or Social Network visitors, or Campaign Visitors, or just Mobile visitors, or different content areas across the site etc.  Discover makes it very easy to do this.

Once the spreadsheet is set up, all you need to do is copy the data back into the sheet and you’ve re-run the engagement metric – in about 5 seconds.

Discover just rocks for this real-time, conscious stream of thought, type of analysis.

In summary

There’s lots of different ways to look at engagement, and hopefully, this will help you understand that there is no single metric, and engagement values change based on various lenses.  But, with the above combination of metrics from the very useful paper by Eric T. Peterson, I believe that we’re closer to understanding engagement, which will help us to modify our sites, or target content better, to try to achieve better levels of engagement by those who are below the medians.

As an aside, part two of this post will be about Visitor Scoring, which is a better Interaction Index than the one demonstrated above – and can be used directly in SiteCatalyst reports.  It involves a bit of custom code for your s_code, and a bit of forethought, but easy enough to do…but I’m saving that for a bit later this week.

Drop me a line or comment below with ways that you are measuring visitor engagement.

It often strikes me as strange that people still look at numbers in the aggregate.  Knowing that you get a certain amount of page views, or a certain amount of visitors and so forth, doesn’t really tell you anything.

In order to get some insights of value, things that you can really act on, you need to segment your traffic and conversions.

But so few people really do it, and even fewer do it really well.  And doing it really well can help you in so many different ways, from optimising your site, to optimising your marketing spend.

People are different; they do things differently.  They interact with your site differently.  Broadly speaking, online, people fall into segments, and segmentation can help you to understand those differences and give you a way to identify those people in the future, so that you can hopefully optimise the experience for them – whether it be through more relevant content, landing page optimisation, or keyword reinforcement etc.

The segmentation challenge

Part of the challenge in segmenting is that you don’t really know what you’re looking for, until you’ve found it.  And “it” is something that’s interesting.  Interesting enough that you can do something with it.  Interesting enough that it stands out from the crowd.

The problem with report based segmentation is that you create a segment (might be something along the lines of traffic from organic search) and you then have to wade your way through the static reports to figure out if there’s anything interesting in them.  In most cases, you will fail to find anything interesting, or usable, because your segment (organic search) is still too broad.

Segmentation is an ad-hoc activity

In reality, finding interesting segments is really an ad-hoc thing.  Interesting things tend to be found through trial and error; and report-based segmentation doesn’t make it easy at all, as you typically can’t dive into data and look at things from different angles.

One of the key tools in the Adobe suite of products that we, at Murdoch Uni use, is Omniture Discover (or more correctly Adobe Discover – powered by Omniture).  From it’s name, you should get that you can “discover” things, through ad-hoc segmentation.  Ad-hoc being segmentation on-the-fly.  Don’t see anything interesting?  Try something else.  Then, when you’ve found something interesting, you can save the segment that meets that same criteria and apply it to your reports in SiteCatalyst, so that you can monitor it over time.

Basic Rules

There’s some basic rules of segmentation though:

1. Segments need to be measurable and identifiable
2. Segments need to be accessible and actionable
3. Segments need to be large enough to be profitable

If your interesting segments don’t meet all three criteria, there’s a good chance that they’re not “interesting” segments.  In other words, a good chance you can’t really do anything with/to them.

Using Discover, we can take a look at traffic to the courses section of our site and do some ad-hoc segmentation to see if anything interesting comes out.  Taking a look at traffic in the aggregate, we see the following:

Which doesn’t really tell us much…  We got 1.67 million page views in a certain time period.

So let’s segment that by Visitor Loyalty – looking at First Time Visitors, versus Repeat Visitors, to see if there’s anything there.  Using Discover, you just drag over the segment First Time Visitor and Loyal Visitor onto the workspace and you instantly get the same metrics for those two additional “pre-configured” segments.

But that still doesn’t really tell us much, other than the obvious…first time visitors look at more pages than repeat visitors, probably because repeat visitors know where the content is they are looking for and can get to them quicker than browsing around.

Now lets segment that by traffic that comes to us from Google Organic search.  We just drill down on the section Courses, and look at our campaign traffic from Google Organic Search, which then segments everything.

Still doesn’t really tell us too much – they appear to be just a subset of the overall segment, behaving in a similar manner.  So we still can’t do too much with that information.

So we’ll segment by time spent, to see if there’s anything interesting in that.  Again, we just drill down on the Google segment showing Time Spent per Visit.

Ah, now that’s interesting…

There’s a high incidence of First Time Visitors from Google who spend between 1-5 minutes on our course pages, whereas, Repeat Visitors tend to spend 30-60 minutes (excluding those that spend 10-30 minutes).

Ok, so what…

Well, we can do something with that.

For instance, we can try to engage 1st Time Visitors from Google more…try to get fewer visitors to spend 1-5 minutes, maybe through cross-promoting content,  maybe through different calls to action.

We could use Test and Target to behaviourally target content at them.  Or we could test different page layouts, or different content altogether, or we could test different promo-modules.

There’s really no end to what we might want to try to do, armed with this information and the capability (and desire) to do something about it.

Segmenting your channel spend

Another way you should be using segmentation is to look at channel effectiveness.  Using ad-hoc segmentation you can see, for example, how many people convert from different segments. 

Let’s assume that you have a Display Campaign running.  You’re running display ads across many network sites, which are all driving clicks to your site. 

You’ll of course know you’re spend (if not, you should do!).

Let’s assume that you are looking for a conversion of some type – it might be a lead, or a sale.  Doesn’t really matter in this example.

And so you’ll ultimately be able to calculate cost per lead, or cost per sale.

However, your cost per lead or cost per sale will vary substantially by different segments.  Three key things that make up that variance are:

1. Sub-segment rates
2. Sub-segment bounce rates
3. Sub-segment conversion rates

Let’s assume that in this example our sub-segments are 1st Time vs. Loyal Visitors and that we get a 60/40 split.

From a bounce standpoint, your bounce rates might be 40% vs. 20% (by sub-segment)

Now factor in that only some of those remaining visitors will decide to start to convert, but they won’t all convert.  Let’s assume that 30% of 1st Time Visitors start to convert, and 40% of loyal visitors start to convert (start to buy something, or start to sign up for something).

Then you factor in your conversion rates.  Lets assume that only 35% of 1st time visitors actually convert, versus 40% for loyal visitors.

Those segments and rates are purely an example, but are feasible given the audience segment.

The following depicts what the above calculations translate into, if you start with a million impressions, on a CPM based display campaign.

Notice the Cost Per Conversion by Segment…$66.37 versus $32.57.

That’s twice as much to get a new customer to convert than a loyal customer. 

So, what can you do…?

How about trying to reduce the bounce rate for first time visitors.  If you can drop it to 20%, you reduce your acquisition cost to $49.67 (from $66.37).  And then you can try to increase your conversion start and completion rates for that specific sub-segment, which will all help towards reducing the cost of acquisition.

Discover Discover

Segmentation in Discover is incredibly easy.  But it doesn’t do only that – it allows you to visualize your traffic and conversions as well.  If you’re just using SiteCatalyst, you really should consider getting Discover – it will, I guarantee, blow your socks off!  It will give you access to view your data in ways that you have only imagined (and wished for).  It will save you time, and probably money in the long run.  It will help you vertically through campaigns and horizontally across campaigns.  (The graphic header to this blog is actually a visualisation of traffic across our website).

Segmentation is the key to success

Segmentation will help you out in so many different ways – and there’s virtually an unlimited number of ways to segment and get valuable insights on your user behaviour.

But the really important thing though, is what you do with it all. 

Once you’ve got the insights, don’t be afraid to try to improve things.  It’s measurable, actionable, and profitable!