Isolating the outcomes of change

Working in digital analytics gives you access to a huge amount of data which with the wrong mindset can cause problems and complaints of drowning. But viewed positively and intelligently it allows you access to huge amounts of insight. Sometimes though, in our eagerness to measure improvements after making changes to a site, we can forget that this treasure-trove of data can help us to refine our measurement and filter out underlying influences on our results. For example, say you've recently made some changes to one of your landing pages and the bounce rate appears to fall in the period afterwards. But what if your marketing team lowered the spending on the PPC campaign on the same day, bringing less traffic to the page? How can you strip this out to get the true effect of your changes on the bounce rate?

A nice way of removing the effects of one other variable is a scatter plot graph. It can show that a site's conversion rate is a diminishing function of the amount of paid traffic the site receives - that paid search traffic's quality is inversely related to its quantity. A site can have varying daily levels of conversion with no change to the site itself, dependent entirely on the level of paid traffic it receives.

Graphing different time periods of the site's visits against conversion rate before and after a change to the site and putting a non-linear trend line through these different periods allows you to monitor its performance. If the changes have worked, the curve moves up and to the right - for a given level of visits, the conversion increases. Instead of determining the average conversion rate before and after the change, you can now compare the distance between the two curves, removing the effect of any fluctuation in traffic - i.e. the position of the observations on the curve.

Another option is to graph a page's bounce rate against entrances in a similar manner to the previous conversion rate example and compare the performance of two different pages, or different stages of a campaign. For example, you can compare the inital stage of a new PPC campaign with subsequent months to see how well it's being optimised to improve relevance and reduce bounce rate.

As with many things, there are caveats to this approach. The first example won't give you a specific measurement to determine the improvement in conversion, but it should serve as a pointer as to how much of the change in conversion was attributable to the page change and how much to paid search traffic fluctuations, allowing you to calculate it manually if need be.

Furthermore, it requires a decent quantity of traffic to allow for a statistically valid conversion or bounce rate and a sizeable number of observations at varying levels of traffic to build the curve. This is not so easy in the digital measurement world as sites and pages change regularly.

Finally, this method only allows you to monitor the effect of one other variable on your changes. There is another tool which can be used to explain the effect of more than one variable in a statistically descriptive manner, which is regression analysis, which I hope to cover in a future post.

The Russian spy ring - an analytical parallel

The recent unearthing of a Russian spy ring working in the United States and the facts uncovered about how its members operated made me think of a couple of parallels with the last post about the public's perception of the secretive world of tracking.

On first impressions one would assume that a spy ring working undercover would be run efficiently, gathering a lot of valuable information, and providing critical insight to the spy-master. However, in this case it soon transpired that the spy ring was actually struggling to function, having issues with both technology and communication. The parallels with some (not all!) company analytics functions are quite clear. Perceived by the general public as an all-knowing collection of spooks, in fact some companies are quite disorganised, struggling to determine what insight their managers need, let alone deliver it. Communication and technology issues can quickly get in the way of extracting clean data from which analysis can provide the necessary insight.

Many of the spy ring's issues centred around a desire amongst some for a return to the "good old days" when things were simpler, be it hiding packages in the ground under bottle tops, or using maps with stamps on. Similarly, there is still a reluctance among many companies to fully embrace digital analytics, with concerns about both the quantity and quality of data, or a timidity around entering the social media arena, let alone measuring it.

As I mentioned in the previous post, we can help ourselves by becoming more transparent and making our case more clearly. But having a better understanding around tracking with the visitors to our sites counts for nothing if we're not organised and brave enough to deliver the insight to drive the change the visitors need.