Today is A-level results day in the UK, when high school students find out if they've got the grades to go to university. Traditionally this is the day the media go to town to bash students or standards, depending on your view point. Students become angry that society has deemed them unworthy of the high grades they get, and society finds it hard to believe the ubiquitous A grades that are handed out are really reflective of the students' understanding.
In my mind there are two clear problems with A level results here - the grading classification itself and how students are being taught, and both these issues rear their heads in the web analytics world.
Firstly, the grading. Here in the UK A levels are marked from A to E, with a new A* grade being introduced this year to try and distinguish the really bright students from the bright ones. However, as more and more students receive the better grades as the years pass, it becomes hard to distinguish the brighter students from the bright - the metric isn't transparent. This is a problem that many analysts try to overcome in their reporting. If the KPI doesn't clearly indicate what's going on, it's going to be hard to take action. Knowing that 50% of your visitors viewed 3 or more pages of your site, and are thus "engaged" doesn't help too much. Knowing the distribution of page views per visits allows you to isolate the extreme cases, and determine who's really ploughing through your site compared to those who just view three pages. In the case of A levels, replacing the grading classification with a simple % scoring system would allow universities to see a more accurate reflection of the students abilities, and compare them with others.
Secondly, metric manipulation. A common complaint is that students are being taught to pass exams, rather than being taught to broaden their knowledge of a subject. Complaints abound that first year university students are unable to string a sentence together or display an understanding of basic numeracy, but they do have a lot of A grades to their name. Back in the world of Web Analytics this manipulation often rears its head too, for example on content sites, where articles are split into multiple pages requiring the reader to click on links to view the next page, thus generating extra page views. This not only proves frustrating for the the visitor, but also implies an artificially high level of engagement with the site.
Of course KPIs are essential in the world of web analytics - they're our bread and butter. And whilst we strive to improve our sites through the monitoring of these KPIs, we need to be bear some things in mind. A KPI is useless unless it accurately depicts the outcomes you're trying to monitor. And manipulating metrics is essentially cheating. And as our Mums all told us, when you cheat, you're only cheating yourself.
No comments:
Post a Comment