What's not to love about Voice of Customer (VOC) data? It's so much easier to figure out what a customer was up when they just come out and tell you, instead of looking for patterns in clickstream data.
Yet whilst it's great when your customers tell you why your site sucks, it can be depressing too. Surely it's obvious where that information is, you cry - it's staring you in the face on the home page! But as we all know, the customer's always right, and what's obvious for us won't always be for them: we need to go and fix the problem. Standard stuff. However, we can do more with this information. As well as monitoring how satisfaction and other customer metrics change after we make these changes, we can also track the comments too.
Building categories from the comments made allows you to not only track the improvements you've made but also keep an eye out for future issues. For example, you could create a category based on feedback which suggests the user can't find the information they're looking for. This could allow you to compute a metric which calculates the percentage of "lost" responders. You could pivot these by visit purpose, or traffic source to get an idea of where the issues are. Some tools allow you to integrate your VOC data with your analytics package, so it could be possible to build segments based on these responders, to overlay on your clickstream data. Maybe they can't find it because it's below the fold on their screen size?
Of course, you don't have to just stick with those who "can't find stuff", you could segment by more vitriolic comments, technical problems, basket problems etc. So take a couple of paracetamol, be brave, and explore your VOC data!
Showing posts with label Analytics. Show all posts
Showing posts with label Analytics. Show all posts
Tuesday, January 18, 2011
Tuesday, November 9, 2010
Using browser data as a net sophistication proxy
Today marks Firefox's 6th birthday, and what better way to celebrate than with a blogpost on browser data.
Within web analytics packages there are plenty of metrics and dimensions that describe user behaviour on your site. However, without resorting to external sources it's hard to build a picture of individuals and their characteristics as opposed to the behaviour they exhibit on your site. However, there are data within web analytics packages that can hint at these characteristics. One example of this is the browser breakdown report: a user's choice of browser works as a proxy for their level of internet sophistication.
Historically, we could say with a fair degree of confidence that those who didn't use Internet Explorer were more advanced users of the internet than those who did. In more recent years, although this still rings true it's not as black and white as it used to be, as IE's market share diminishes in light of the general public's increasing awareness of the alternatives. There are of course exceptions to this - those who use multiple browsers, or those using the internet in a work environment where their browser choice is restricted, although this can be overcome. However, in general those who use non-IE browsers are by definition exhibiting preferences that indicate their more sophisticated use of the internet.
This definition of sophistication can be improved by looking at browser versions rather than just browsers themselves. Doing this could give you an indication of how early adopters (those using dev or beta versions of browsers) interact with your site as opposed to luddites (those still on IE6), and gives you more flexibility into how you define people. The downside to this though is you need to keep up to date with your definitions as browser updates now come thick and fast. And, of course, you don't have to stop there with your definitions - adding other dimensions (for example keywords used or keyword count) can further refine them.
Creating segments based on these definitions can open up a lot of insights into your site behaviour and traffic sources. However, you need to bear in mind that it is a proxy, and first and foremost it describes the difference in behaviour of visitors using different browsers - so if you see some weird and wacky behaviour as a result of this, your first port of call should be to check how your site functions for this browser rather than put it down to users being less/more sophisticated than average. That said, with some common sense and imagination you can uncover plenty of interesting stuff using different interpretations of "standard" web analytics dimensions.
Within web analytics packages there are plenty of metrics and dimensions that describe user behaviour on your site. However, without resorting to external sources it's hard to build a picture of individuals and their characteristics as opposed to the behaviour they exhibit on your site. However, there are data within web analytics packages that can hint at these characteristics. One example of this is the browser breakdown report: a user's choice of browser works as a proxy for their level of internet sophistication.
Historically, we could say with a fair degree of confidence that those who didn't use Internet Explorer were more advanced users of the internet than those who did. In more recent years, although this still rings true it's not as black and white as it used to be, as IE's market share diminishes in light of the general public's increasing awareness of the alternatives. There are of course exceptions to this - those who use multiple browsers, or those using the internet in a work environment where their browser choice is restricted, although this can be overcome. However, in general those who use non-IE browsers are by definition exhibiting preferences that indicate their more sophisticated use of the internet.
This definition of sophistication can be improved by looking at browser versions rather than just browsers themselves. Doing this could give you an indication of how early adopters (those using dev or beta versions of browsers) interact with your site as opposed to luddites (those still on IE6), and gives you more flexibility into how you define people. The downside to this though is you need to keep up to date with your definitions as browser updates now come thick and fast. And, of course, you don't have to stop there with your definitions - adding other dimensions (for example keywords used or keyword count) can further refine them.
Creating segments based on these definitions can open up a lot of insights into your site behaviour and traffic sources. However, you need to bear in mind that it is a proxy, and first and foremost it describes the difference in behaviour of visitors using different browsers - so if you see some weird and wacky behaviour as a result of this, your first port of call should be to check how your site functions for this browser rather than put it down to users being less/more sophisticated than average. That said, with some common sense and imagination you can uncover plenty of interesting stuff using different interpretations of "standard" web analytics dimensions.
Tuesday, October 19, 2010
Improving Engagement
Engagement's back on the menu. Eric Peterson gave a webinar recently with a excellent overview of engagement and discussed some new white papers written to measure it. However, due to the ambiguous nature of engagement, these measurement techniques (despite being the best attempt so far) are quite complicated and need to be tailored to the individual website.
Historically we measured engagement using page views per visit, time on site etc, but there was no way to define positive or negative sentiment, or distinguish an engaged visitor from a lost one. Currently companies are trying to bring in other datasets at their disposal to help, such as social and Voice of the Customer, but it's still fuzzy and subjective. We need to spend more time thinking about what engagement is, rather than just thinking of it as "someone who digs my site". To what extent are current methodologies for engagement measurement capturing actual engagement? Are we using the metrics at our disposal to define the concept of engagement as well as measure it?
Ideally we'd define engagement by what the visitors to our sites are thinking whilst carrying out their activities on the site as much as by what they did. But this isn't an ideal world. What we need to avoid doing is defining it by just selecting the metrics we can hold up against them. Broadly, an engaged visitor is likely to view more pages and exhibit an increased propensity to interact with your site whether internally (e.g leaving comments on posts) or externally (e.g. linking to your site). Obviously this again is dependent on the site and site type so can't be defined too tightly, creating the engagement paradox - we have a limited number of valid metrics at our disposal to capture behaviour that is too varied to define accurately. But it gets worse - we also need to bear in mind that visitors are unique and as such will interact differently on a site. Another thing we might need to take into consideration is at what stage they're at on the customer lifecycle:
Whilst we can argue the toss at which stage a visitor would become engaged, we can certainly agree that the latter stages would define engaged visitors. However, visitors in these different latter categories would likely display different types of behaviour, even though they were "engaged". A visitor who's yet to make a purchase but is close to making a decision would behave differently to a loyal multi-buyer. To me this highlights the fact that with the current tools at our disposal it's going to be hard work to build an engagement model anytime soon.
Might this change? Looking to the future, the increased importance of mobile to analytics and its implications for future web behaviour (geolocation) will bring more parameters and data to be used in the calculation of engagement. Whether this will make the calculation of engagement easier or not is debatable. Perhaps this is somewhere that the paid tools can bring some innovation to the market by looking to build an engagement feature into the interface? We're forever hearing the concerns around the amount of data available, but the lack of information coming out of it - this would be a great opportunity to right that wrong.
Historically we measured engagement using page views per visit, time on site etc, but there was no way to define positive or negative sentiment, or distinguish an engaged visitor from a lost one. Currently companies are trying to bring in other datasets at their disposal to help, such as social and Voice of the Customer, but it's still fuzzy and subjective. We need to spend more time thinking about what engagement is, rather than just thinking of it as "someone who digs my site". To what extent are current methodologies for engagement measurement capturing actual engagement? Are we using the metrics at our disposal to define the concept of engagement as well as measure it?
Ideally we'd define engagement by what the visitors to our sites are thinking whilst carrying out their activities on the site as much as by what they did. But this isn't an ideal world. What we need to avoid doing is defining it by just selecting the metrics we can hold up against them. Broadly, an engaged visitor is likely to view more pages and exhibit an increased propensity to interact with your site whether internally (e.g leaving comments on posts) or externally (e.g. linking to your site). Obviously this again is dependent on the site and site type so can't be defined too tightly, creating the engagement paradox - we have a limited number of valid metrics at our disposal to capture behaviour that is too varied to define accurately. But it gets worse - we also need to bear in mind that visitors are unique and as such will interact differently on a site. Another thing we might need to take into consideration is at what stage they're at on the customer lifecycle:
Whilst we can argue the toss at which stage a visitor would become engaged, we can certainly agree that the latter stages would define engaged visitors. However, visitors in these different latter categories would likely display different types of behaviour, even though they were "engaged". A visitor who's yet to make a purchase but is close to making a decision would behave differently to a loyal multi-buyer. To me this highlights the fact that with the current tools at our disposal it's going to be hard work to build an engagement model anytime soon.
Might this change? Looking to the future, the increased importance of mobile to analytics and its implications for future web behaviour (geolocation) will bring more parameters and data to be used in the calculation of engagement. Whether this will make the calculation of engagement easier or not is debatable. Perhaps this is somewhere that the paid tools can bring some innovation to the market by looking to build an engagement feature into the interface? We're forever hearing the concerns around the amount of data available, but the lack of information coming out of it - this would be a great opportunity to right that wrong.
Tuesday, September 28, 2010
The future of web analytics
As the web analytics industry matures its future is still uncertain. In this post I'll have a look at some of the questions that we'll need to answer soon.
There's been a lot of takeover activity in the industry of late, and two of the main analytics commentators, John Lovett and Eric Peterson, have written about the mergers and what they mean of late. Now only WebTrends is left as an independent tool, zigging whilst the others zag, and IBM is splitting at the seams with its three recent acquisitions, to make 23 in the last four years. Whilst they may not have released plans to close any of these products down, one has to wonder if this will be beneficial for the industry, with the potential for stiffling innovation, which is what the industry craves at the moment.
Then there's the bifurcation in tools debate. Some maintain that to do truly sophisticated analytics you need a more powerful (and expensive) tool, with the likes of Google Analytics being left to the marketers. Whilst Google doesn't offer an visitor-level intelligence tool as some of the paid solutions do, no-one can deny the progress the tool has made in recent years. But will it ever truly catch up and end the bifurcation of tools (and is it in their commercial interests to do so)? And what about Google's future itself - how reliant on its parent is Google Analytics? With Facebook and others starting to take on the big G, and its recent attempts to enter the social arena backfiring, the company's future isn't guaranteed, and its analytics package isn't at the top of its list of priorities. However, the tool has no clear competitors in the free arena, with Yahoo! Analytics maintaining it's non-mainstream enterprise-only position for the foreseeable future. What if a new (suitably big) entrant decided to get in on the free game? Perhaps Microsoft might reconsider their exit from this field? If they or another did, it could force Google to further up its game.
Finally, there's the soft side of analytics - the skills required to do the job. Currently a knowledge of statistics isn't that important; being business savvy or having coding knowledge more helpful. But what if other factors change? Will the rise of mobile require a more technical person to understand the intracies of it? Will the rise of intuitivly-designed and easy-to-implement analytical packages mean that company knowledge and the intepretation of these numbers becomes more important to bring context and relevancy? As sites evolve and improve through competition and analytical insight, will visitor-level tools become obligatory for commercial sites? If they do, the skills set of the web analyst will have to expand too.
There's been a lot of takeover activity in the industry of late, and two of the main analytics commentators, John Lovett and Eric Peterson, have written about the mergers and what they mean of late. Now only WebTrends is left as an independent tool, zigging whilst the others zag, and IBM is splitting at the seams with its three recent acquisitions, to make 23 in the last four years. Whilst they may not have released plans to close any of these products down, one has to wonder if this will be beneficial for the industry, with the potential for stiffling innovation, which is what the industry craves at the moment.
Then there's the bifurcation in tools debate. Some maintain that to do truly sophisticated analytics you need a more powerful (and expensive) tool, with the likes of Google Analytics being left to the marketers. Whilst Google doesn't offer an visitor-level intelligence tool as some of the paid solutions do, no-one can deny the progress the tool has made in recent years. But will it ever truly catch up and end the bifurcation of tools (and is it in their commercial interests to do so)? And what about Google's future itself - how reliant on its parent is Google Analytics? With Facebook and others starting to take on the big G, and its recent attempts to enter the social arena backfiring, the company's future isn't guaranteed, and its analytics package isn't at the top of its list of priorities. However, the tool has no clear competitors in the free arena, with Yahoo! Analytics maintaining it's non-mainstream enterprise-only position for the foreseeable future. What if a new (suitably big) entrant decided to get in on the free game? Perhaps Microsoft might reconsider their exit from this field? If they or another did, it could force Google to further up its game.
Finally, there's the soft side of analytics - the skills required to do the job. Currently a knowledge of statistics isn't that important; being business savvy or having coding knowledge more helpful. But what if other factors change? Will the rise of mobile require a more technical person to understand the intracies of it? Will the rise of intuitivly-designed and easy-to-implement analytical packages mean that company knowledge and the intepretation of these numbers becomes more important to bring context and relevancy? As sites evolve and improve through competition and analytical insight, will visitor-level tools become obligatory for commercial sites? If they do, the skills set of the web analyst will have to expand too.
Wednesday, September 15, 2010
The future of the web and the implications for its measurement
Guessing the future of the web is a game that everyone likes to play, but because it's still early days and the web is still volatile, the forecaster normally ends up looking silly. But I'm going to carry on and make some predictions anyway and have a think about what this means for those in the web analytics community.
Although it typically accounts for less that 10% of visits to websites, it's obvious that with the rise of smartphones and tablets the mobile share of web browsing will soon dominate. Handset manufacturers will focus on the devices' power and bringing in new functionality, with more consumer focus being put on the operating system and software. What new apps will be developed? Currently geolocation is the flavour of the month, with Facebook joining in the fun. In my experience the geography data fed back through web analytics tools is not that accurate. With the future of the internet becoming more reliant on geography, this might be something we need to improve. Whilst there are again privacy implications for the improving this accuracy, imagine the potential for finding out where your customers are when browsing your site. Or being able to integrate check-in data with your web analytics data?
With issues around privacy, complaints about its applications and other negative publicity Facebook's seems to be peaking. We regularly hear about the risks of putting all your marketing eggs in the Facebook basket, but does the web analytics industry not risk doing the same thing? However, companies are now not only building relationships with customers within these arenas on fan pages, but monitoring what's said about them within these arenas outside of their fan pages. Sentiment analysis is one area with real potential but is reliant on still-developing artificial intelligence. To me, this is closely linked with the struggle to get to Web 3.0 - the semantic web, where we try to bring more meaning to the content on the internet, and build relationships between data and datasets. Thinking about how we struggle to manage our data now, and how the providers struggle to present it makes it clear how much progress will be required to accurately manage, link and present this new era of data. I think that one of the largest challenges facing our industry is how this is managed, owned and presented in the future, perhaps second only to how we address our current privacy issues.
The majority of the world is still coming to terms with the implications of the "always available" internet, and its potential for increased communication whether it be for good or ill. As the authorities attempt to track illicit online behaviour, there's a growing confusion between monitoring civilians' behaviour and data, and web analytics. Whilst I believe we need to step up to this and nip it in the bud, I would hope that eventually the public takes a more relaxed attitude to tracking, in the way that they do to store loyalty cards, for example. We also need to consider the implications of a generation growing up with the internet as its main resource for entertainment and education. It doesn't seem beyond the realms of possiblility for future companies to be set up to help 18 year olds change their identity and escape their permanently documented youthful transgressions, as hypothesised by Eric Schmidt recently. Might there be an opportunity for building tools to help individuals track their online presence? Whilst the ease with which students can now research information will help them discover more, on the downside it's now easier to plagiarise other's work for assignments and communicate in exams. The recently introduced Tynt Tracer may be further developed to help track illicit copying in this framework, with analytics agencies being set up to monitor other people's sites rather than their own in order to optimise their site.
Indeed, it's this side of analytics that I think we need to be considering now. Whilst the model of working for a company to help optimise their website is the current standard, perhaps we should start thinking outside the box. The internet is now central to more and more people's lives, and whilst this will continue to drive this existing model for those in the web analytics industry, there are opportunities to be had for working on other sites. These could be governmental, educational, looking at analysing external sites for a company, as suggested above, or indeed working for individuals, perhaps to measure the data held on them by other companies? All in all, this shows that the web analytics industry should be kept quite busy keeping up with developments on the internet.
I'd love to hear your thoughts on this - am I way off mark? Have I missed something which you think we need to consider?
Although it typically accounts for less that 10% of visits to websites, it's obvious that with the rise of smartphones and tablets the mobile share of web browsing will soon dominate. Handset manufacturers will focus on the devices' power and bringing in new functionality, with more consumer focus being put on the operating system and software. What new apps will be developed? Currently geolocation is the flavour of the month, with Facebook joining in the fun. In my experience the geography data fed back through web analytics tools is not that accurate. With the future of the internet becoming more reliant on geography, this might be something we need to improve. Whilst there are again privacy implications for the improving this accuracy, imagine the potential for finding out where your customers are when browsing your site. Or being able to integrate check-in data with your web analytics data?
With issues around privacy, complaints about its applications and other negative publicity Facebook's seems to be peaking. We regularly hear about the risks of putting all your marketing eggs in the Facebook basket, but does the web analytics industry not risk doing the same thing? However, companies are now not only building relationships with customers within these arenas on fan pages, but monitoring what's said about them within these arenas outside of their fan pages. Sentiment analysis is one area with real potential but is reliant on still-developing artificial intelligence. To me, this is closely linked with the struggle to get to Web 3.0 - the semantic web, where we try to bring more meaning to the content on the internet, and build relationships between data and datasets. Thinking about how we struggle to manage our data now, and how the providers struggle to present it makes it clear how much progress will be required to accurately manage, link and present this new era of data. I think that one of the largest challenges facing our industry is how this is managed, owned and presented in the future, perhaps second only to how we address our current privacy issues.
The majority of the world is still coming to terms with the implications of the "always available" internet, and its potential for increased communication whether it be for good or ill. As the authorities attempt to track illicit online behaviour, there's a growing confusion between monitoring civilians' behaviour and data, and web analytics. Whilst I believe we need to step up to this and nip it in the bud, I would hope that eventually the public takes a more relaxed attitude to tracking, in the way that they do to store loyalty cards, for example. We also need to consider the implications of a generation growing up with the internet as its main resource for entertainment and education. It doesn't seem beyond the realms of possiblility for future companies to be set up to help 18 year olds change their identity and escape their permanently documented youthful transgressions, as hypothesised by Eric Schmidt recently. Might there be an opportunity for building tools to help individuals track their online presence? Whilst the ease with which students can now research information will help them discover more, on the downside it's now easier to plagiarise other's work for assignments and communicate in exams. The recently introduced Tynt Tracer may be further developed to help track illicit copying in this framework, with analytics agencies being set up to monitor other people's sites rather than their own in order to optimise their site.
Indeed, it's this side of analytics that I think we need to be considering now. Whilst the model of working for a company to help optimise their website is the current standard, perhaps we should start thinking outside the box. The internet is now central to more and more people's lives, and whilst this will continue to drive this existing model for those in the web analytics industry, there are opportunities to be had for working on other sites. These could be governmental, educational, looking at analysing external sites for a company, as suggested above, or indeed working for individuals, perhaps to measure the data held on them by other companies? All in all, this shows that the web analytics industry should be kept quite busy keeping up with developments on the internet.
I'd love to hear your thoughts on this - am I way off mark? Have I missed something which you think we need to consider?
Tuesday, August 24, 2010
Improving self-improvement: a call for open-source education
It was simpler in the olden days: you bought the tool, read the documentation and voila! you'd taught yourself web analytics (well, almost). Now, to be at the top of your game in this business you need to be continuously learning. One of the many great reasons for working in the web analytics industry is its rate of development, with lots of new tools and techniques being introduced, and different thoughts abound on how to do the job properly.
There are a variety of learning resources available to the budding web analyst. There are many blogs in the web analytics field debating the latest issues, giving advice and suggesting new ways to tackle old problems (I've listed a few in my blog list to the right, if you're interested). There are also forums, books, and white papers provided by consultancies and vendors, catering to those in the visual learner category. For those auditory learners there are a number of podcasts out there (see also banner to the right). This then leaves the excitingly named kinesthetic learners who learn by doing, which sounds like the perfect opportunity to plug the Analysis Exchange.
So there are a number of places a web analyst can rely on to keep up-to-date with what's going on. But this puts me in mind of the former US Secretary of Defense, Donald Rumsfeld, talking about known unknowns. These resources are all great at helping you find out information about things you know that exist and you know little about, the known unknowns. But what about the unknown unknowns? How can you get a definitive list of everything that a web analyst should know, to determine if you're on top of it all? I believe that this is something that the WAA is missing. Whilst they currently have the syllabus for the WAA Certification, publishing a list of the areas involved in "Web Analytics" might help define the role of the web analyst better, and help them in their efforts to define themselves too. It could help build a coherent self-referenced set of pages on the intracies of web analytics, with suggestions for the metrics and reports to use for given scenarios. Whilst there's plenty of information out there providing overviews of web analytics and the tools to use, quite often the advice contained glosses over the details, or is one-dimensional, failing to mention other related reports or analyses that could be carried out. This then would become the definitive site for a web analytics education.
The science of web analytics has been around for a while now. So why hasn't this "open-source" educational resource been created yet? Being spoon-fed the information isn't the best way to learn - what good, curious web analyst would want to learn this way? With the current web analytics sphere being very tools-centric it becomes harder to share information as silos develop. And there's also an element of self-interest. Handing out the information on a plate loses business for practioners; it also spoils book sales.
And yet, I still feel that open-source education is the way to move forwards. Whilst the web analytics industry has been around for a while, it's still not mature. The public doesn't trust it, and whilst the majority of companies have at least one web analytics solution on their site, there's little evidence it's being used to its potential, with only the largest or bravest allowing their online strategy to be steered by it. In order to deal with this, we need to grow the number of individuals with the necessary knowledge to become advocates, dedicated to analysing their website on a full time basis. Restricting the ease with which they can learn is a short-termist approach - we need to think about the long term. By growing an army of trained web analysts, the case for the benefits of analytics can be made to those businesses still too small or immature to have made the transition, transforming companies from being satisfied with a list of their top 10 pages to ones competing on analytics, to paraphrase Stephane Hamel's OAMM model. As a critical mass of sites that truly use analytics is reached, the remainder will have to engage or die. Competition breeds improvements in techniques and ideas. Then, as the world learns that sophisticated web analytics requires sufficient resourcing, the opportunity for consulting services and more specialist knowledge will grow, and the availability of information on the internet becomes irrelevant. No-one teaches themselves accountancy - they hire an accountant. By sharing now, we can create the demand for tomorrow.
There are a variety of learning resources available to the budding web analyst. There are many blogs in the web analytics field debating the latest issues, giving advice and suggesting new ways to tackle old problems (I've listed a few in my blog list to the right, if you're interested). There are also forums, books, and white papers provided by consultancies and vendors, catering to those in the visual learner category. For those auditory learners there are a number of podcasts out there (see also banner to the right). This then leaves the excitingly named kinesthetic learners who learn by doing, which sounds like the perfect opportunity to plug the Analysis Exchange.
So there are a number of places a web analyst can rely on to keep up-to-date with what's going on. But this puts me in mind of the former US Secretary of Defense, Donald Rumsfeld, talking about known unknowns. These resources are all great at helping you find out information about things you know that exist and you know little about, the known unknowns. But what about the unknown unknowns? How can you get a definitive list of everything that a web analyst should know, to determine if you're on top of it all? I believe that this is something that the WAA is missing. Whilst they currently have the syllabus for the WAA Certification, publishing a list of the areas involved in "Web Analytics" might help define the role of the web analyst better, and help them in their efforts to define themselves too. It could help build a coherent self-referenced set of pages on the intracies of web analytics, with suggestions for the metrics and reports to use for given scenarios. Whilst there's plenty of information out there providing overviews of web analytics and the tools to use, quite often the advice contained glosses over the details, or is one-dimensional, failing to mention other related reports or analyses that could be carried out. This then would become the definitive site for a web analytics education.
The science of web analytics has been around for a while now. So why hasn't this "open-source" educational resource been created yet? Being spoon-fed the information isn't the best way to learn - what good, curious web analyst would want to learn this way? With the current web analytics sphere being very tools-centric it becomes harder to share information as silos develop. And there's also an element of self-interest. Handing out the information on a plate loses business for practioners; it also spoils book sales.
And yet, I still feel that open-source education is the way to move forwards. Whilst the web analytics industry has been around for a while, it's still not mature. The public doesn't trust it, and whilst the majority of companies have at least one web analytics solution on their site, there's little evidence it's being used to its potential, with only the largest or bravest allowing their online strategy to be steered by it. In order to deal with this, we need to grow the number of individuals with the necessary knowledge to become advocates, dedicated to analysing their website on a full time basis. Restricting the ease with which they can learn is a short-termist approach - we need to think about the long term. By growing an army of trained web analysts, the case for the benefits of analytics can be made to those businesses still too small or immature to have made the transition, transforming companies from being satisfied with a list of their top 10 pages to ones competing on analytics, to paraphrase Stephane Hamel's OAMM model. As a critical mass of sites that truly use analytics is reached, the remainder will have to engage or die. Competition breeds improvements in techniques and ideas. Then, as the world learns that sophisticated web analytics requires sufficient resourcing, the opportunity for consulting services and more specialist knowledge will grow, and the availability of information on the internet becomes irrelevant. No-one teaches themselves accountancy - they hire an accountant. By sharing now, we can create the demand for tomorrow.
Thursday, August 19, 2010
A Levels and KPIs - an analogy
Today is A-level results day in the UK, when high school students find out if they've got the grades to go to university. Traditionally this is the day the media go to town to bash students or standards, depending on your view point. Students become angry that society has deemed them unworthy of the high grades they get, and society finds it hard to believe the ubiquitous A grades that are handed out are really reflective of the students' understanding.
In my mind there are two clear problems with A level results here - the grading classification itself and how students are being taught, and both these issues rear their heads in the web analytics world.
Firstly, the grading. Here in the UK A levels are marked from A to E, with a new A* grade being introduced this year to try and distinguish the really bright students from the bright ones. However, as more and more students receive the better grades as the years pass, it becomes hard to distinguish the brighter students from the bright - the metric isn't transparent. This is a problem that many analysts try to overcome in their reporting. If the KPI doesn't clearly indicate what's going on, it's going to be hard to take action. Knowing that 50% of your visitors viewed 3 or more pages of your site, and are thus "engaged" doesn't help too much. Knowing the distribution of page views per visits allows you to isolate the extreme cases, and determine who's really ploughing through your site compared to those who just view three pages. In the case of A levels, replacing the grading classification with a simple % scoring system would allow universities to see a more accurate reflection of the students abilities, and compare them with others.
Secondly, metric manipulation. A common complaint is that students are being taught to pass exams, rather than being taught to broaden their knowledge of a subject. Complaints abound that first year university students are unable to string a sentence together or display an understanding of basic numeracy, but they do have a lot of A grades to their name. Back in the world of Web Analytics this manipulation often rears its head too, for example on content sites, where articles are split into multiple pages requiring the reader to click on links to view the next page, thus generating extra page views. This not only proves frustrating for the the visitor, but also implies an artificially high level of engagement with the site.
Of course KPIs are essential in the world of web analytics - they're our bread and butter. And whilst we strive to improve our sites through the monitoring of these KPIs, we need to be bear some things in mind. A KPI is useless unless it accurately depicts the outcomes you're trying to monitor. And manipulating metrics is essentially cheating. And as our Mums all told us, when you cheat, you're only cheating yourself.
In my mind there are two clear problems with A level results here - the grading classification itself and how students are being taught, and both these issues rear their heads in the web analytics world.
Firstly, the grading. Here in the UK A levels are marked from A to E, with a new A* grade being introduced this year to try and distinguish the really bright students from the bright ones. However, as more and more students receive the better grades as the years pass, it becomes hard to distinguish the brighter students from the bright - the metric isn't transparent. This is a problem that many analysts try to overcome in their reporting. If the KPI doesn't clearly indicate what's going on, it's going to be hard to take action. Knowing that 50% of your visitors viewed 3 or more pages of your site, and are thus "engaged" doesn't help too much. Knowing the distribution of page views per visits allows you to isolate the extreme cases, and determine who's really ploughing through your site compared to those who just view three pages. In the case of A levels, replacing the grading classification with a simple % scoring system would allow universities to see a more accurate reflection of the students abilities, and compare them with others.
Secondly, metric manipulation. A common complaint is that students are being taught to pass exams, rather than being taught to broaden their knowledge of a subject. Complaints abound that first year university students are unable to string a sentence together or display an understanding of basic numeracy, but they do have a lot of A grades to their name. Back in the world of Web Analytics this manipulation often rears its head too, for example on content sites, where articles are split into multiple pages requiring the reader to click on links to view the next page, thus generating extra page views. This not only proves frustrating for the the visitor, but also implies an artificially high level of engagement with the site.
Of course KPIs are essential in the world of web analytics - they're our bread and butter. And whilst we strive to improve our sites through the monitoring of these KPIs, we need to be bear some things in mind. A KPI is useless unless it accurately depicts the outcomes you're trying to monitor. And manipulating metrics is essentially cheating. And as our Mums all told us, when you cheat, you're only cheating yourself.
Monday, August 2, 2010
Multichannel: the digital holy grail or a poisoned chalice?
It is the received wisdom that the 360 degree view of the customer should be the ultimate goal of all marketing functions. By combining their customers' online and offline activities the company can monitor how frequently and through which channels these customers are touched by or touch them, ascertain how the customers respond to incentives, and use this information to determine how best to market to them. But should this be the goal of analytical functions? Might it actually be a waste of time?
There are a number of implementation issues that have to be overcome for this sort of project to avoid turning from the digital holy grail to a poisoned chalice. Firstly, accuracy. Whilst we've been told before that accuracy isn't so important in web analytics, this isn't the case when it comes to combining multiple databases - having an all-singing-all-dancing database means nothing if your data's not up-to-scratch. Your offline data needs to be regularly cleaned to remove deceased customers and update addresses, otherwise your finely-honed marketing campaign will be flawed from the start. In addition to this there's the problem of linking the offline to the online records - if any of these are inaccurate, then you're going to have problems merging the two. So there's little point executing this project unless you're satisfied that databases can be accurately linked. Finally, this isn't cheap. This of course is no reason not to implement a scheme providing the ROI warrants it. But can that be guaranteed, given some of these potential pitfalls?
Once the implementation problems are out of the way, you face your next hurdle (which ideally you'd have considered before the implementation). This relates to the type of industry your company sits in. A multichannel programme is going to be of little use to you if your customers don't purchase from you that frequently, which could be the case if you're in a very competitive industry, or one where the purchase frequency is low (cars, for example). If your customers don't purchase from you through any channel that frequently, this causes two problems for you: the statistical robustness of the data is weakened and the accuracy of the data you hold on them is likely to deteriorate between purchases, as customers identifying information changes. Finally, and crucially, this data only deals with the purchasers through your differing channels, and obviously won't pick up those who fail to purchase, assuming that visitors to your site only login during the purchasing process. It provides little help for converting prospects by determining why they didn't purchase from you.
And this leads me on to my main point - assuming you've got this far and set up an accurate multichannel database with a 360-degree view of your customers who are purchasing frequently enough from you to keep it all together, what next? It won't have happened overnight, and it won't have been free. Is being able to determine that customer A has responded better to a direct mail than an email campaign, whereas customer B only purchases online really going to provide huge insight? Obviously insight is there to be found, and it may be of benefit to some companies. But wouldn't a well constructed and segmented email campaign have already told you that customer A didn't respond well? Looking at it in terms of the opportunity cost of implementing such a costly and timely scheme, isn't there something more productive you could have done instead? For example, looking at segmenting your email campaigns more effectively, and analysing their on-site behaviour compared to other visitors might give you an improved response rate to your campaigns.
It seems to me that this is the end result of a marketer's fantasy gone mad, with little thought for the practical realities of its implementation and shortfalls. It's part of a familiar scenario: we've got too much data, and we're struggling to deliver true, clear insight. So what do we do to solve the problem? Bring in more data, or try and link existing datasets in an attempt to find it. But often there is no identifiable answer, and actually customer A asked his partner who happens to be customer B to buy it for him online. Focussing on the basics could provide as much ROI implementing a mulitchannel solution.
Again, let me reiterate, I'm not saying there's nothing to be gained from doing this; just that it's being sold as solving all our problems and being some sort of Utopian ideal, when in fact it takes a lot of time and money to implement, and there are still many key questions out there that remain unanswered.
There are a number of implementation issues that have to be overcome for this sort of project to avoid turning from the digital holy grail to a poisoned chalice. Firstly, accuracy. Whilst we've been told before that accuracy isn't so important in web analytics, this isn't the case when it comes to combining multiple databases - having an all-singing-all-dancing database means nothing if your data's not up-to-scratch. Your offline data needs to be regularly cleaned to remove deceased customers and update addresses, otherwise your finely-honed marketing campaign will be flawed from the start. In addition to this there's the problem of linking the offline to the online records - if any of these are inaccurate, then you're going to have problems merging the two. So there's little point executing this project unless you're satisfied that databases can be accurately linked. Finally, this isn't cheap. This of course is no reason not to implement a scheme providing the ROI warrants it. But can that be guaranteed, given some of these potential pitfalls?
Once the implementation problems are out of the way, you face your next hurdle (which ideally you'd have considered before the implementation). This relates to the type of industry your company sits in. A multichannel programme is going to be of little use to you if your customers don't purchase from you that frequently, which could be the case if you're in a very competitive industry, or one where the purchase frequency is low (cars, for example). If your customers don't purchase from you through any channel that frequently, this causes two problems for you: the statistical robustness of the data is weakened and the accuracy of the data you hold on them is likely to deteriorate between purchases, as customers identifying information changes. Finally, and crucially, this data only deals with the purchasers through your differing channels, and obviously won't pick up those who fail to purchase, assuming that visitors to your site only login during the purchasing process. It provides little help for converting prospects by determining why they didn't purchase from you.
And this leads me on to my main point - assuming you've got this far and set up an accurate multichannel database with a 360-degree view of your customers who are purchasing frequently enough from you to keep it all together, what next? It won't have happened overnight, and it won't have been free. Is being able to determine that customer A has responded better to a direct mail than an email campaign, whereas customer B only purchases online really going to provide huge insight? Obviously insight is there to be found, and it may be of benefit to some companies. But wouldn't a well constructed and segmented email campaign have already told you that customer A didn't respond well? Looking at it in terms of the opportunity cost of implementing such a costly and timely scheme, isn't there something more productive you could have done instead? For example, looking at segmenting your email campaigns more effectively, and analysing their on-site behaviour compared to other visitors might give you an improved response rate to your campaigns.
It seems to me that this is the end result of a marketer's fantasy gone mad, with little thought for the practical realities of its implementation and shortfalls. It's part of a familiar scenario: we've got too much data, and we're struggling to deliver true, clear insight. So what do we do to solve the problem? Bring in more data, or try and link existing datasets in an attempt to find it. But often there is no identifiable answer, and actually customer A asked his partner who happens to be customer B to buy it for him online. Focussing on the basics could provide as much ROI implementing a mulitchannel solution.
Again, let me reiterate, I'm not saying there's nothing to be gained from doing this; just that it's being sold as solving all our problems and being some sort of Utopian ideal, when in fact it takes a lot of time and money to implement, and there are still many key questions out there that remain unanswered.
Friday, July 16, 2010
Isolating the outcomes of change
Working in digital analytics gives you access to a huge amount of data which with the wrong mindset can cause problems and complaints of drowning. But viewed positively and intelligently it allows you access to huge amounts of insight. Sometimes though, in our eagerness to measure improvements after making changes to a site, we can forget that this treasure-trove of data can help us to refine our measurement and filter out underlying influences on our results. For example, say you've recently made some changes to one of your landing pages and the bounce rate appears to fall in the period afterwards. But what if your marketing team lowered the spending on the PPC campaign on the same day, bringing less traffic to the page? How can you strip this out to get the true effect of your changes on the bounce rate?
A nice way of removing the effects of one other variable is a scatter plot graph. It can show that a site's conversion rate is a diminishing function of the amount of paid traffic the site receives - that paid search traffic's quality is inversely related to its quantity. A site can have varying daily levels of conversion with no change to the site itself, dependent entirely on the level of paid traffic it receives.
Graphing different time periods of the site's visits against conversion rate before and after a change to the site and putting a non-linear trend line through these different periods allows you to monitor its performance. If the changes have worked, the curve moves up and to the right - for a given level of visits, the conversion increases. Instead of determining the average conversion rate before and after the change, you can now compare the distance between the two curves, removing the effect of any fluctuation in traffic - i.e. the position of the observations on the curve.
Another option is to graph a page's bounce rate against entrances in a similar manner to the previous conversion rate example and compare the performance of two different pages, or different stages of a campaign. For example, you can compare the inital stage of a new PPC campaign with subsequent months to see how well it's being optimised to improve relevance and reduce bounce rate.
As with many things, there are caveats to this approach. The first example won't give you a specific measurement to determine the improvement in conversion, but it should serve as a pointer as to how much of the change in conversion was attributable to the page change and how much to paid search traffic fluctuations, allowing you to calculate it manually if need be.
Furthermore, it requires a decent quantity of traffic to allow for a statistically valid conversion or bounce rate and a sizeable number of observations at varying levels of traffic to build the curve. This is not so easy in the digital measurement world as sites and pages change regularly.
Finally, this method only allows you to monitor the effect of one other variable on your changes. There is another tool which can be used to explain the effect of more than one variable in a statistically descriptive manner, which is regression analysis, which I hope to cover in a future post.
A nice way of removing the effects of one other variable is a scatter plot graph. It can show that a site's conversion rate is a diminishing function of the amount of paid traffic the site receives - that paid search traffic's quality is inversely related to its quantity. A site can have varying daily levels of conversion with no change to the site itself, dependent entirely on the level of paid traffic it receives.
Graphing different time periods of the site's visits against conversion rate before and after a change to the site and putting a non-linear trend line through these different periods allows you to monitor its performance. If the changes have worked, the curve moves up and to the right - for a given level of visits, the conversion increases. Instead of determining the average conversion rate before and after the change, you can now compare the distance between the two curves, removing the effect of any fluctuation in traffic - i.e. the position of the observations on the curve.
Another option is to graph a page's bounce rate against entrances in a similar manner to the previous conversion rate example and compare the performance of two different pages, or different stages of a campaign. For example, you can compare the inital stage of a new PPC campaign with subsequent months to see how well it's being optimised to improve relevance and reduce bounce rate.
As with many things, there are caveats to this approach. The first example won't give you a specific measurement to determine the improvement in conversion, but it should serve as a pointer as to how much of the change in conversion was attributable to the page change and how much to paid search traffic fluctuations, allowing you to calculate it manually if need be.
Furthermore, it requires a decent quantity of traffic to allow for a statistically valid conversion or bounce rate and a sizeable number of observations at varying levels of traffic to build the curve. This is not so easy in the digital measurement world as sites and pages change regularly.
Finally, this method only allows you to monitor the effect of one other variable on your changes. There is another tool which can be used to explain the effect of more than one variable in a statistically descriptive manner, which is regression analysis, which I hope to cover in a future post.
Wednesday, June 9, 2010
Applying statistical rigour to web analytics reporting
Web analytics is all about making decisions from the data. But how can you be sure of the quality of the data you investigate, and the recommendations you provide from it? Whilst the numbers may be accurate and reflect what happened on your site thanks to a successful tagging implementation, are they statistically significant? Furthermore, once you've uncovered what's fluke and what's not, how can you illustrate this succinctly in your reporting?
Unfortunately, with a few limited exceptions, vendors don't provide any indication of the robustness of the data in their consoles. Wouldn't it be great if, for a selected time period, you could segment your site and see that although there's a marked difference in conversion (or metric of preference) that it's only significant at the 50% level? Or, alternatively, what appears to be only a small difference in bounce rate is actually statistically significant? Until that day comes though, you need to be able to do it yourself. Avinash has written about on a couple of related topics a while back - applying statistical limits to reporting and an overview of statistical significance. In a more recent post, Anil Batra highlights the importance of not rushing to pick the winning page from an A/B test. And in the last few days, Alec Cochrane has written a great piece on how to improve the statistical significance of a web analytics data-based model.
There are plenty of statistics tests out there, with different datasets and situations that call for them, but for the purpose of this post I'll focus on just two both of which are listed here amongst others.
The two proportion z test compares whether the specified proportions of two samples are statistically different from one another.
This test can be applied to a number of reports within web analytics, but its main use would be for comparing the click-through rate or response rate of two campaigns to determine whether one is conclusively better than the other. The beauty of this test is that it only requires four values - the two proportions (%s) and the two sample sizes, and as such can be calculated without use of a spreadsheet.
The second test is the two-sample t-test which determines whether the means of two samples are statistically different from each other for the two given sample sizes and sample standard deviations.
By requiring the standard deviations of both samples, this result takes more time to compute by requiring the user to download the series data in question. This test has a variety of uses, for example comparing whether the different average values of a given metric for two segments are statistically different, or perhaps looking at the same data series before and after an external event takes place to determine whether it has had a statistically significant effect on the data or not.
Now that you're confident that you know which results are statistical flukes and which ones aren't, how do you go about illustrating this in your reporting? One option would be to include the t test results and significance levels in your reporting, but this is likely to clutter your reports as well as potentially confuse the reader. A neater way might be to colour code the values to illustrate their confidence level if you're happy to introduce different text colours to your report. For time series data you can add the mean, and upper and lower bounds to a graph, to show which peaks and troughs merit further investigation.
Of course, once you've come up with a clear and succinct way of displaying this statistical information, you still need to explain it to your stakeholders, not all of whom will have a background in statistical analysis. Demonstrating the robustness of the data to them and how the varying levels of robustness are determined will not only provide extra confidence in the decisions you're recommending from the data, but illustrate the importance of asking constructive questions of the data, rather than slavishly following what the data suggests at first glance.
Images from Wikipedia.org
Unfortunately, with a few limited exceptions, vendors don't provide any indication of the robustness of the data in their consoles. Wouldn't it be great if, for a selected time period, you could segment your site and see that although there's a marked difference in conversion (or metric of preference) that it's only significant at the 50% level? Or, alternatively, what appears to be only a small difference in bounce rate is actually statistically significant? Until that day comes though, you need to be able to do it yourself. Avinash has written about on a couple of related topics a while back - applying statistical limits to reporting and an overview of statistical significance. In a more recent post, Anil Batra highlights the importance of not rushing to pick the winning page from an A/B test. And in the last few days, Alec Cochrane has written a great piece on how to improve the statistical significance of a web analytics data-based model.
There are plenty of statistics tests out there, with different datasets and situations that call for them, but for the purpose of this post I'll focus on just two both of which are listed here amongst others.
The two proportion z test compares whether the specified proportions of two samples are statistically different from one another.
This test can be applied to a number of reports within web analytics, but its main use would be for comparing the click-through rate or response rate of two campaigns to determine whether one is conclusively better than the other. The beauty of this test is that it only requires four values - the two proportions (%s) and the two sample sizes, and as such can be calculated without use of a spreadsheet.
The second test is the two-sample t-test which determines whether the means of two samples are statistically different from each other for the two given sample sizes and sample standard deviations.
By requiring the standard deviations of both samples, this result takes more time to compute by requiring the user to download the series data in question. This test has a variety of uses, for example comparing whether the different average values of a given metric for two segments are statistically different, or perhaps looking at the same data series before and after an external event takes place to determine whether it has had a statistically significant effect on the data or not.
Now that you're confident that you know which results are statistical flukes and which ones aren't, how do you go about illustrating this in your reporting? One option would be to include the t test results and significance levels in your reporting, but this is likely to clutter your reports as well as potentially confuse the reader. A neater way might be to colour code the values to illustrate their confidence level if you're happy to introduce different text colours to your report. For time series data you can add the mean, and upper and lower bounds to a graph, to show which peaks and troughs merit further investigation.
Of course, once you've come up with a clear and succinct way of displaying this statistical information, you still need to explain it to your stakeholders, not all of whom will have a background in statistical analysis. Demonstrating the robustness of the data to them and how the varying levels of robustness are determined will not only provide extra confidence in the decisions you're recommending from the data, but illustrate the importance of asking constructive questions of the data, rather than slavishly following what the data suggests at first glance.
Images from Wikipedia.org
Tuesday, May 25, 2010
The Times Paywall - Not such a bad idea?
The Times of London (owner Mr R. Murdoch) recently announced it would be introducing a paywall to its two sites, due to go live at the start of June, and has been receiving a lot of coverage about it. Previous experiments with paywalls have yielded poor results, and there has been a lot of discussion, especially on social networking sites about how setting up a walled garden and attemping to work against the principle of an open internet will ultimately hurt Mr Murdoch. Furthermore, keeping content behind a paywall would not only limit the availability of the site's news, but also its reporters. Users of Twitter would no longer be able to link to their work, and see it be shared across the internet - their brand as well as their paper's would be curtailed.
Looking at it in purely monetary terms, this is clearly a beneficial move. Mr Murdoch needs to ensure that
VaCaR + PVa > VbCbR
where a denotes after and b before the paywall is introduced, V is the number of visitors, C the display ad clickthrough rate, R the revenue per click, and P the subscription price.
Making some assumptions about the share of visits to visitors before and after the paywall change, taking the visitor values for The Times with the forecast 95% fall in traffic afterwards, and assuming that the clickthrough rate will improve following the introduction of a paywall suggests that unless revenue per click is more than approximately £26, this will generate more revenue for News Corp. This demonstrates that this move will clearly benefit them (unless anyone knows of an ad that regularly yields that sort of revenue per click!). Of course the cost to the brand in terms of damaged reputation, reduced visability and possible loss of staff is not so easy to calculate.
In terms of analytics, though, things get more interesting. Reducing the amount of traffic to the site in this way should tighten the audience profile - effectively removing the drifters, and bringing the online profile of visitors closer to the offline one. This then should enable more effective advertising, appealing to the more shared interests of the new profile. Also, the new traffic should be more engaged with the site, generating more page views per visit, and thus more opportunity for clicking on the ads, as well as helping any behavioural advertising the company may be using. And of course in terms of the site itself rather than the advertising, the more engaged traffic should improve the conversion funnels the site has, such as engaging with their live chat facility.
I'll be interested to hear what News Corp has to say about the effects of its paywall from this perspective in a month or so!
Looking at it in purely monetary terms, this is clearly a beneficial move. Mr Murdoch needs to ensure that
VaCaR + PVa > VbCbR
where a denotes after and b before the paywall is introduced, V is the number of visitors, C the display ad clickthrough rate, R the revenue per click, and P the subscription price.
Making some assumptions about the share of visits to visitors before and after the paywall change, taking the visitor values for The Times with the forecast 95% fall in traffic afterwards, and assuming that the clickthrough rate will improve following the introduction of a paywall suggests that unless revenue per click is more than approximately £26, this will generate more revenue for News Corp. This demonstrates that this move will clearly benefit them (unless anyone knows of an ad that regularly yields that sort of revenue per click!). Of course the cost to the brand in terms of damaged reputation, reduced visability and possible loss of staff is not so easy to calculate.
In terms of analytics, though, things get more interesting. Reducing the amount of traffic to the site in this way should tighten the audience profile - effectively removing the drifters, and bringing the online profile of visitors closer to the offline one. This then should enable more effective advertising, appealing to the more shared interests of the new profile. Also, the new traffic should be more engaged with the site, generating more page views per visit, and thus more opportunity for clicking on the ads, as well as helping any behavioural advertising the company may be using. And of course in terms of the site itself rather than the advertising, the more engaged traffic should improve the conversion funnels the site has, such as engaging with their live chat facility.
I'll be interested to hear what News Corp has to say about the effects of its paywall from this perspective in a month or so!
Subscribe to:
Posts (Atom)
