Now, there’s a hot topic.  Measuring engagement.  One of the most widely debated topics in web analytics.

What is engagement and how do we measure it?

Engagement, unfortunately, is not derived from a single measure.  It’s not time on site.  It’s not how many pages they viewed.  It’s not bounce rates and it’s not about conversions.

Engagement is about a lot of things.  What is an engaged visitor and how do you measure engagement?

“Visitor Engagement is an estimate of the depth of visitor interaction against a clearly defined set of goals.” Eric T. Peterson and Joseph Carrabis.

A while ago, I came across their paper through Web Analytics Demystified, entitled “Measuring the Immeasurable: Visitor Engagement”.  While I won’t go into it in any detail, I will suggest that you read it, as it’s the background of this post.

The premise of the paper is that visitor engagement is made up of 7 different metrics, and expressed through one formula:

where:

Engagement can be expressed as the average of the sum of indexes, across specific segments, according to a:

1. Click Depth Index – which captures the contribution of page and event views
2. Duration Index – capturing the contribution of time spent on site
3. Recency Index – which captures the visitor’s “visit velocity”—the rate at which visitors return to the web site over time
4. Loyalty Index- the level of long-term interaction the visitor has with the brand, site, or product(s)
5. Brand Index – the apparent awareness of the visitor of the brand, site, or product(s)
6. Feedback Index – qualitative information including propensity to solicit additional information or supply
7. Interaction Index – visitor interaction with content or functionality designed to increase level of Attention

According to Eric T. Peterson, “Visitor Engagement is a function of the number of clicks (Ci), the visit duration (Di), the rate at which the visitor returns to the site over time (Ri), their overall loyalty to the site (Li), their measured awareness of the brand (Bi), their willingness to directly contribute feedback (Fi) and the likelihood that they will engage in specific activities on the site designed to increase awareness and create a lasting impression (Ii).”

When applied at the visitor level, on a per-visitor basis, they combine to form a pretty good proxy for visitor engagement.

Having read the paper, I was intrigued, and decided to use Discover to implement this…to some pretty insightful results.

A summary of the indexes

Click Depth Index

The percentage of your overall audience that has a minimum threshold of an acceptable number of page views per session.  If you see that on average, visitors “convert” after viewing at least 5 pages, then your minimum threshold would be 5 pages per visit.

Duration Index

The percentage of your overall audience that has a minimum threshold of an acceptable amount of time on site per session.  If you see that on average, visitors “convert” after spending at least 10 minutes on your site, then your minimum threshold would be 10 minutes.

Recency Index

The percentage of your overall audience that returns and converts within an acceptable amount of time (generally days).  If you notice that most visitors convert between 1 and 10 days, then you’d be looking for visitors with a return frequency of <= 10 days.

Loyalty Index

The percentage of your overall audience that has a repeat visit frequency in excess of a minimum threshold.  For example, if you notice that many visitors convert after visiting your site more than three times, then your threshold would be a visit count of at least 3.

Brand Index

The percentage of your overall audience that comes to your site either directly, or through branded search terms.

Feedback Index

The percentage of your overall audience that completes feedback on your site, or participates in rating or reviewing content, or commenting on blogs.

Interaction Index

The percentage of your overall audience that interacts with specific content on your site, or engages in an activity on your site.  There are no thresholds for this index – they are simply counts of.

Note: while you can count pre-defined activities on your site, it is better to score visitor interaction.  I’ll be doing a post on visitor scoring shortly.

Using Discover

Discover was built for this!  It’s very easy to create segments within Discover and apply them across various views to gain insight.

Firstly what we did was to look at some of the thresholds to understand what our “Anonymous” segment of traffic does.  Our anonymous segment is made of non-student and non-staff traffic, which we already have segments for in both Discover and SiteCatalyst.

We figured out what our minimum page views per session should be, the average duration, frequency of visit etc, by looking at them from a conversion standpoint…i.e. how many pages does a converter see, on average, before converting.

Once we’d done that, our 7 segments were easy to define as follows:

1. Click Depth Index – Visitor container, Path length > 10
2. Duration Index – Visitor container, Seconds spent per visit > 1800 (30 minutes)
3. Recency Index – Visitor container, Return Frequency <= 7-14 days
4. Loyalty Index – Visitor container, Visit number >= 2
5. Brand Index – Visitor container, Organic Search Keyword contains “Murdoch” or Visit without referrer
6. Feedback Index – (we don’t use this one)
7. Interaction Index – Visitor container, any of the following events: Lead Complete, Application Complete, Form Complete, Tool Name

The 8th segment was All Visits.  In each case, we used the Visitors metric to view the number of visitors that were part of each index.

If we view this against referring sites, what we end up with is the number of visitors that match each segment rule:

Export to Excel

What we need to do now is export the data to Excel to do the averages and generate the final engagement value.

Simply select the first item “None”, click Ctrl+A for select all, then click Ctrl+C for copy.

Open Excel, and paste the raw data into a new sheet.

Then it’s simply a matter of calculating one columns percentage as a percentage of the All Visits – Visitors column.

Once you’ve done that for each column, you have the indexes for each segment.  Now you just average all of the indexes to get an engagement metric.

For example, we see in the above that Direct Traffic, “none” in the above report, has an overall engagement value of 23%.  But if we look at the other columns, we also see that they are at the median value on Click Depth, whereas traffic from deewr.gov.au is well above the median.

A couple of interesting things have also been highlighted in the above, for example, traffic from Google Singapore is actually far more engaged than traffic from Google Australia – now that’s interesting.

Of course, you should always look at engagement via larger segments, for example, by Campaign, by Site, by Time of Day, Day of Week etc.

While Tuesday comes out overall for a better engaged visitor, I’ve highlighted other interesting things, such as on Saturday and Sunday visitors click more, but more visitors spend time on Saturdays.  During the week is better for branded search term visitors, and Wednesdays seems to be better overall for key interactions.

Multiple Sites

If you have multiple sites, such as microsite etc, you might want to check engagement across them to see if they are dramatically different, so you can then begin to try to understand why.

In the above calculation, I’ve removed the Feedback and Interaction indexes from the calculation, as they would skew the results.  It’s interesting that while the main University site has an engagement index of 23.67, versus a median of 20.43%, sites like Mobile and Maps have a very high engagement value.

Segmentation

Once you’ve got the basic engagement working and you’re looking at things from the overall perspective, you can then easily begin to look at engagement by different segments.

In Discover, you can create segments on-the-fly and apply them across your other segments, by simply dragging the new segment onto the filtered workspace. For example, we segment the above by Anonymous visitors, after we’ve built the overall segments.  We can do the same for Converting Visitors, or Social Network visitors, or Campaign Visitors, or just Mobile visitors, or different content areas across the site etc.  Discover makes it very easy to do this.

Once the spreadsheet is set up, all you need to do is copy the data back into the sheet and you’ve re-run the engagement metric – in about 5 seconds.

Discover just rocks for this real-time, conscious stream of thought, type of analysis.

In summary

There’s lots of different ways to look at engagement, and hopefully, this will help you understand that there is no single metric, and engagement values change based on various lenses.  But, with the above combination of metrics from the very useful paper by Eric T. Peterson, I believe that we’re closer to understanding engagement, which will help us to modify our sites, or target content better, to try to achieve better levels of engagement by those who are below the medians.

As an aside, part two of this post will be about Visitor Scoring, which is a better Interaction Index than the one demonstrated above – and can be used directly in SiteCatalyst reports.  It involves a bit of custom code for your s_code, and a bit of forethought, but easy enough to do…but I’m saving that for a bit later this week.

Drop me a line or comment below with ways that you are measuring visitor engagement.

One of the fundamental things you need to understand about Omniture SiteCatalyst is the difference between an s.prop and an eVar, and just what events are and when to set them.  They are at the heart of the product and provide the ability to customise it to suit your business needs.

If you don’t understand the difference, you’re going to be in a world of pain, and left dazed and confused.

This is, understandably, the most confusing thing to new SiteCatalyst users, and they take a bit of getting used to, especially when you start to combine them all together, but once you understand them, you’ll be on your way to generating custom ones that can really provide insight.

The clickable illustration on the right started as a Sunday afternoon musing, but then extended into a somewhat chaotic depiction of props, evars, events, campaigns, products and so forth – of which I’ve tried to explain a little more simply below.

s.prop definition

These are custom traffic variables.  They are used to count the number of times that certain values are sent into SiteCatalyst.  With the latest release of SiteCatalyst, you get 75 custom traffic variables to play with.

They are not persistent – meaning that once they get counted in SiteCatalyst they get forgotten.  Nothing else can get attributed to them.

Traffic data includes visits, visitors, page views, sections, sub sections, internal search terms, user type (segments) etc.  You can also enable pathing on these custom variables to understand the path that users take from prop to prop.

If you want to breakdown two traffic variables by one another, such as Pages by Browsers, then you must ensure that both variables are set on the same page.  The correlation report only shows instances where two things occurred at the same time.

If you wanted to understand internal search terms that eventually get the user to a form, traffic props are not the way to go.  For that, you need to use an eVar.  And in many cases, when you set a custom traffic prop, you’ll also want to set a custom eVar too.

Mostly you’ll just pass a single value into an s.prop…maybe the name of the sub section, or the name of a tool, or the type of user currently logged in, or the category of content etc.  There’s another type of s.prop, which is called a list s.prop.  List props take a delimited list of values, and then they’re split out into separate line items in SiteCatalyst.

Bear in mind that list props cannot be correlated…despite the fact they’re broken out by SiteCatalyst into their individual elements.

eVar definition

These are called conversion variables and are generally set on different pages.  Again, you get 75 of these too.

They are usually used to tie success events back to the last value that was stored in the eVar.  By definition, these are persistent, and you control, through the admin, how long they remain persistent (visit or timeframe or when something happens like a success event), and how to allocate a success event to them (most recent value is the most common setting).

eVar relationships

eVars can be related (or broken down) by one another.  There are two ways to achieve this – basic subrelations and full subrelations: 

• Full subrelations can be broken down by any other eVar that has either full or basic subrelations enabled.
• Basic subrelations can only be broken down by an eVar that has full subrelations enabled.
• The third type is no subrelations and they cannot be broken down by anything.

By default, campaigns and products are enabled with full subrelations out of the box.

Consider this:

If you want to know which internal search terms lead to the most form submissions, and which search terms lead to tool usage on your site, then Search Terms needs to have Full Sub Relations enabled on it.  That way you can break down Search Terms by any eVar, and the other eVars (such as Form Name and Tool Name) can also be broken down by Search Term.

Crossing eVars with sProps

You can’t.  You just, plain, can’t.  Accept this and move on.

You cannot cross traffic values with conversion values.  They don’t mix.  As soon as you remember that, and plan for that, you’re doing ok.  That’s one of the reasons you generally set an eVar every time you set a traffic prop. You can correlate two traffic props together (browser and subsection for exmaple), and you can subrelate two conversion variables together, but in SiteCatalyst, you can’t cross props and eVars. 

If you’ve got Discover, well that’s a different story.  You can pretty much cross anything you like with anything you like, against segments and times on the fly, to your hearts content!  Seriously, if you don’t have Discover, get in touch with your account manager for a demo – you will NOT look back.  And if you have a lot of eVars that you want full subrelations on, then you’re a prime candidate for Discover, not to mention if you’re using ASI slots for segmentation reasons – or you just can’t get that report that you’re looking for.  Discover will do it for you.  Period.

Events

Success events are counts of specific things that occur on your site, usually things like a form view, or a registration, or a login, or an application.  Success events are set and tied to an eVar.  Your reports will show the number of times that success events have been set against the specific values on the eVars.

Normal success events, such as when a registration form is viewed and then completed, takes two success events, one for the view and the other for the completion. 

So, let’s assume you have a registration form.  You want to know how many people view the form and how many people submit the form.

On the registration form page, you’d set the following:

s.events = "event1"; // this is your form view event
s.eVar1 = "Registration Form"; // this is the name of the form

Then, on the thank you page, you’d set the following:

s.events = "event2"; // this is your form complete event
s.eVar1 = "Registration Form"; // this is the name of the form

Notice that eVar1 is set to the same name in both instances, but has different success events set.

In SiteCatalyst, you’d create eVar1, named something like “Forms” which will automatically create a new report for you called Forms.  You’d view the Custom Conversion > Forms report, being eVar1, add in the metrics Form Views and Form Completes, and it will show you how many form views have happened (event1) and how many form completes have happened (event2) during the specified time period.

Special Events

Then there are special events; product views, shopping cart view, open, add, remove and checkout, and finally a purchase.  These are generally used for measuring products purchased and shopping cart activity.

A product example

So let’s assume that you are a financial institution and have information on various credit cards as well as an application form for each type of card.  You want to know how many times the card information has been viewed, as well as applications started and submitted, across a multi-page application process.  Additionally, you want to track the credit limit applied for on the card application.

On the credit card information page, you set the special event prodView (and it’s also best practice to set another success event as the prodView event is only available within the product reports).

So, you could use the following:

s.events = "prodView,event3"; // product view and a success event
s.products = ";Credit Card XYZ"; // this is the name of the product
s.eVar2 = "Credit Card XYZ";

The product string usually takes many more parameters, but as we’re only setting it for a product view, we only need to set the name of the product in the product string. 

The other parameters, that are required when something is purchased, are as follows:

• Category (legacy – leave blank so that you can use Classifications to better group products)
• Product Name
• Number of Units
• Total Revenue from Units
• Events and eVars (but we’ll save those as they’re more complicated but can be used for things like tracking shipping costs or discounts etc)

Note that you MUST start the product string with a semi-colon if you are not using the category.  You don’t generally use the first parameter, Category, because the best way to do that is to use classifications to group products together.

So, now you’ve got the product views measured, each time someone goes to the product page, event3 will be set against eVar2, and prodView will be set against the product Credit Card XYZ.  ProdView is one of those special event counters.

To get the Application Start metric, on the first page of the application you set the following:

s.events = "event4"; // application start event
s.eVar2 = "Credit Card XYZ";

To get the product purchase, the revenue, the amount and the successful submission, on the application thank you page you’d set the following:

s.events = "purchase,event5"; // purchase and application submitted event
s.eVar2 = "Credit Card XYZ";
s.products = ";Credit Card XYZ;1;10000"; // product;units;total revenue
s.purchaseID = "123456789"; // unique application code

The events set include the special event “purchase” and event5, in this case, the application submitted event.

eVar2 is the name of the product for the conversion reports.

Products lists the name of the product, the number of units sold (1) and the revenue (10000 – in this case its the credit limit applied for).

The purchaseID would need to be a unique identifier, possibly the application number, so that SiteCatalyst can de-dupe any entries.

Now you have your product reports populated with the number of units sold and the total credit limits applied for, being the revenue amount (if that’s how you track revenue).

To sum it up

Props are traffic variables. eVars are conversion variables.  Events are things that happen on your site and are tied to conversion variables.  You can’t cross the two together, but can cross props and you can subrelate eVars.  Oh, and you need to get Discover (‘coz it rocks). Did I mention that already?

Here’s another really simple customisation that you can and should do as part of your basic implementation, which helps you to further understand attribution.

Attribution is probably one of the hardest and most contested measurements available…which “thing” led your customer to do something.

Campaign Stacking

In SiteCatalyst, that’s called Stacking.  And there’s really no end to things you can “stack”.

Stacking essentially means that you store different values in the order that they occurred – and you can do it across multiple visits as well.  So, for example, if you’re stacking campaign codes, you’ll see in your reports:

campaign1> campaign2 > campaign3 > etc

From there, you can tie the success event, such as Lead, or Purchase, to the individual variations and have some visibility around how many times they interact with different elements before they convert.

Granted, it doesn’t generate the cleanest of reports, but there is insight to be gained from it.

At Murdoch Uni, we’ve done that for a number of different views:

1. Campaigns (the obvious one)
2. Internal Campaigns (the second obvious one)
3. Courses viewed (see the order of courses viewed before a conversion)
4. Lead Type (as we have multiple ways to generate leads, we want to know which tools they go through)

An example of campaign stacking report, broken down (due to full sub-relations being enabled on the eVar) is:

The above report has been filtered on “>” to only show those that have interacted with more than one campaign.

If you’re an avid reader of my blog, you’ll remember an earlier post of “People who liked this, also liked”, where I showed how to achieve the same for product views.

The above shows the unique ID of the course (or product), the number of times we saw that same sequence, the number of leads and apps generated from that sequence.  As I indicated in the previous post – our users don’t exhibit the behaviour of viewing multiple courses…

If we look at which tools (aka forms or flash devices) they interact with and become a lead through (as they can do it through multiple tools, we see the following:

Mmm, well, not a lot of data in this one as we’ve only just turned this one on – but you get the idea.

Getting smarter…

Another way that you could do it fairly easily, with a bit more custom code, is to stack not only campaign codes, but the referring domain, typein, paid search, organic search, etc instead of just leaving it at known campaign codes. 

And so, to the nitty gritty

You use the plugin s.crossVisitParticipation.

The following is an example for traditional campaign code stacking:

/* Campaign Stacking */
s.eVar36=s.crossVisitParticipation(s.campaign,'s_evar36','90','10','>','');

The values in the plugin are as follows:

1^st = value to pass into the stack.

2^nd = Name of the cookie to set that stores the stack

3^rd = Days the cookie will last for

4^th = Number of things to store in the stack

5^th = Delimiter in stack

Can’t for the life of me remember what the final element is though (but whatever it is, we don’t use it).

You’ll also need an eVar to put it into, which you should add in via your Admin – enable full sub-relations if you can.

As with all of these, you should contact Client Care or Consulting to make sure you can get the plugin and that you implement it correctly.

Final thoughts

If you’ve got Test and Target too, you’re laughing!

You can target for re-engagement if they’ve met certain criteria (or haven’t met a certain criteria).  So, if someone’s got campaign code 1 and campaign code 2, but hasn’t converted, show them a different offer.

Discover also allows you to further gain insights by analysing results and traffic against this eVar.

That’s it really.  Happy stacking.

In my previous post on People who liked this, also liked…, I put forward an idea how to generate “related” products of interest, based on what users were looking at, which could then be automated and re-published back to a site, based on Omniture data.

We implemented this in our s_code, largely as a test, to see what the results would be like, and to further enable proof of concept testing when we exported the data.

Well, we’ve had an interesting observation as a result of this (we haven’t yet fully implemented the re-publishing part) and I thought it was worthy of a quick posting.

Our expectation of user behaviour was that they look at multiple courses, during their first visit – in essence, they would browse courses looking at what was on offer, reading about the various differences between them.

It turns out that is not the case.

If we look at Course views over a certain period we can see what are the most popular courses (and not so popular).  What I have shown here is a list of the top 15 courses viewed over a specific time frame.  So we can see from this that Vet Science was the most viewed course over time, but it doesn’t show whether this was the “first choice” or the “second choice”.  It doesn’t show whether this was viewed during the initial visit or a follow up visit.

So, this doesn’t help in our People who liked this, also liked…scenario.

In putting in place our Course Stacking code, we are now able to see the order in which courses are typically viewed in.

Bear in mind that we have implemented this as a solution that we can export and then re-integrate into our site, so we have purposely used Course ID, not the name of the course.

The result within SiteCatalyst, is that the report isn’t easy to read…but I’ll attempt to explain it.

The Course View Stacking column contains the Course ID’s in the order that they were viewed.  So for example, the first entry “39>146″ tells us that there were 133 instances where a user started with viewing course 39, then went on to view course 146.

Utilizing Course Stacking, there are an enormous amount of combinations that will ultimately show up in this report.  Just in the short time this has been going, it has already generated 6,275 combinations.

Of course, this could be other products, product categories etc.

So we can get these into perspective, and not use just ID’s, Course 39 is Law (Four-Year Degree) (LLB), and Course 146 is Law (Three-Year Degree) Juris Doctor (LLB).

So, what we’re seeing is that there have been 133 instances of visitors starting with the four year Law degree, then going on to look at the three year Law degree.

A bit further down the report in position 9, you’ll also see “39>146>39″.  This is a different stream to the above.  In this stream, there have been 47 instances where visitors have gone from the 4yr degree, to the 3yr degree and back to the 4yr degree – which is different to the above, where they went from the 4yr to the 3yr, and have not looked at anything else since.

This is the reason why we can get so many combinations.

Ok, so overall, that seems to make sense – users would look at other courses.

Notice the percentage (of overall) is very low.  This activity only occurred 0.5% of the time.  This is due to the amount of time we’ve had this running.

However, remember our original assumption – users will look for a course, then browse to other courses, during their first visit.

Out of curiosity I exported the data from Omniture, segmented by New vs Repeat visitors, so see whether the behaviour changed.  The question I wanted to answer was “do users typically look at more than one course [product] during their first visit”.

I was surprised by the result.

I actually used a different product to analyze the results here.  The above chart shows the number of different courses viewed (x-axis) and the number of times that occurred.  New visitors are blue, repeat visitors are orange.

It’s split roughly 50/50 for people who view 1 course, but it’s certainly the largest category, inferring that the majority of visitors come to the site and are engaged with one course, which is also good news, suggesting that they are satisfied with the content of the course.

Now look at the weightings for those who view more than one course.  There are clearly more repeat visitors who view 3 or more courses, and that really spikes at the end (10 or more courses are grouped together).

This suggests that our original assumption was wrong.  First time users, on balance, view only one course.  But when they come back, they view either one course, or more than one course – suggesting that they begin to browse courses only on their repeat visit.

This is very interesting and insightful.  We can use this to our advantage.  We can target content to repeat visitors who have viewed a course previously – either prompting them back into their original course, or present them with “related” courses.

We can also try to better cross-promote courses on their first visit.

This was an unexpected insight that came out of this analysis, but to us, very valuable information that can be used.

I was chatting with one of our School Deans today about various results and he posed the question “Is it possible to see which courses people viewed after seeing one course?”.  His interest was based on the fact that the user doesn’t always purchase the “most frequently visited course”.  They often view one thing, but end up purchasing something else, and our reporting doesn’t highlight that behaviour.

Now, that got me thinking…that’s probably pretty common behaviour.  So how can we make that visible?

Pathing is common

Of course, it’s easy to show page pathing (which pages are viewed before and after a certain page), section pathing (similar but for a section), but pathing isn’t available across multiple visits (for the obvious reasons).  Traffic pathing is available on s.props, so as long as you report something into an s.prop, you can generate paths to/from it.  Paths are very valuable to see where a user goes after visiting a specific item such as a page, or how they got to a specific page.

However, the problem arises when you want to see something across multiple visits.

We’ve just had a similar problem with multi-visit campaign results, where the success event was being attributed to the latest campaign id, which wasn’t neccessarily what we expected.  In our case, due to the sales cycle being long (typically 1-3 months), many visits will occur and the user won’t always come in with the same campaign code.

For example, we might send them an email which drives them to the site.  The user engages, finds out what they need, but doesn’t convert.  They then come back a few days or weeks later by either typing in our web address directly, or come in through a search engine.  In either case, the success event (if they convert) would be attributed to the latest campaign, for example, Google or Direct/Typein (as we also have a VISTA rule).

Enter Campaign Stacking…

So, to provide some visibility to this activity, we worked with our consultant who recommended we implement Campaign Stacking, which, through the use of a cookie, appends a different campaign code (if they have one) to any previous one.

So, in the above example, we now have reports which show conversions by campaign combination.  We accomplished this by setting up a new eVar and writing a cookie (through an s_code plugin) appending the next campaign code to a previous campaign code.

Now we should be able which campaign combinations are driving conversions, over multiple visits.

Now stay with me…

I’ll bet we can do the same thing to understand product view combinations over multiple visits, leading to conversion.

In our case, a product is a course, but no reason this couldn’t work for any product category.  In our case, we don’t want to see which course “pages” they visited (we have that through course page pathing).  We want to see course pathing across multiple visits (or the same visit).

By setting an eVar with the name of the course, and using the same methodology as above, we should be able to get a view on this activity and user behaviour.

In theory, we should then be able to export the data and generate promo-type content that says “People who liked this course, also liked these courses…”

That will then help us to cross-promote “related” courses – not what we think are related, but what users are thinking are related.  Do that on an automated, daily basis and you really start to apply some value for the user.

That’s one of the great things about Omniture – flexibility to do this.

Guess what I’ll be trying over the next few days…I’ll update this one over time, if we get it working.