Friday, July 16, 2010

Isolating the outcomes of change

Working in digital analytics gives you access to a huge amount of data which with the wrong mindset can cause problems and complaints of drowning. But viewed positively and intelligently it allows you access to huge amounts of insight. Sometimes though, in our eagerness to measure improvements after making changes to a site, we can forget that this treasure-trove of data can help us to refine our measurement and filter out underlying influences on our results. For example, say you've recently made some changes to one of your landing pages and the bounce rate appears to fall in the period afterwards. But what if your marketing team lowered the spending on the PPC campaign on the same day, bringing less traffic to the page? How can you strip this out to get the true effect of your changes on the bounce rate?

A nice way of removing the effects of one other variable is a scatter plot graph. It can show that a site's conversion rate is a diminishing function of the amount of paid traffic the site receives - that paid search traffic's quality is inversely related to its quantity. A site can have varying daily levels of conversion with no change to the site itself, dependent entirely on the level of paid traffic it receives.

Graphing different time periods of the site's visits against conversion rate before and after a change to the site and putting a non-linear trend line through these different periods allows you to monitor its performance. If the changes have worked, the curve moves up and to the right - for a given level of visits, the conversion increases. Instead of determining the average conversion rate before and after the change, you can now compare the distance between the two curves, removing the effect of any fluctuation in traffic - i.e. the position of the observations on the curve.

Another option is to graph a page's bounce rate against entrances in a similar manner to the previous conversion rate example and compare the performance of two different pages, or different stages of a campaign. For example, you can compare the inital stage of a new PPC campaign with subsequent months to see how well it's being optimised to improve relevance and reduce bounce rate.

As with many things, there are caveats to this approach. The first example won't give you a specific measurement to determine the improvement in conversion, but it should serve as a pointer as to how much of the change in conversion was attributable to the page change and how much to paid search traffic fluctuations, allowing you to calculate it manually if need be.

Furthermore, it requires a decent quantity of traffic to allow for a statistically valid conversion or bounce rate and a sizeable number of observations at varying levels of traffic to build the curve. This is not so easy in the digital measurement world as sites and pages change regularly.

Finally, this method only allows you to monitor the effect of one other variable on your changes. There is another tool which can be used to explain the effect of more than one variable in a statistically descriptive manner, which is regression analysis, which I hope to cover in a future post.

No comments:

Post a Comment

Real Time Web Analytics
Feedback Form
Website Feedback