The Devil Is In The Data!

The Devil Is In The Data!

There are analytics and metrics these days to cover pretty much each and every segment of your business. Some of these metrics are freely available (if you know where to look) and others are purpose built and intrinsically linked to your business objectives. The trick is knowing exactly how much faith to put in to metrics and when…

There was a case very recently (read: Monday 25 June, 2012) that put this dilemma into sharp perspective.  In short, a blogger decided to take it upon himself to review the internet listenership numbers that 2x popular South African internet radio stations were publicizing. He claimed that the internet radio stations were reporting inflated figures and that the real figures which he had access to were considerably more humble. The full story is here for your reading pleasure.

Of course, now everyone is wondering, “What numbers can you trust?”. And the answer is quite simple : None of them some of the time and all of them the rest of the time!

Let’s break this up a little bit more into bite-sized chunks and look at 2 of the faces of reporting that can lead to the most glaring of inconsistencies:

  1. Business owned reports vs. externally generated reports
  2. Data interpretation

 Business owned vs externally generated reports

An example I like to use here is Google Analytics vs. Server data reports. There are many differences between each of these and the way in which they collect data that you are virtually never going to get these numbers looking 100% the same. There is always going to be a variance and it could be from something as small as the tracking code not loading completely on the page, to something larger like an internal DNS problem that loops or tracks incorrectly.

 Data interpretation

Again using the example about; many reports can speak term different data the same way; in other words a website analytics report can talk about visitors and not differentiate whether it is unique visitors or total visitors. This confusion in reporting can lead to some very different numbers being quoted. This gets compounded when date ranges are included and data is compared over these differing data ranges.

There are literally countless business situations where the data that you are using is critical to the decision you need to take; whether it is reporting on listener numbers to justify your rate card; reporting back on ROI for a marketing campaign or even more intrinsic business operating data.

So how can we safeguard ourselves when looking at and interpreting data?

  1. Name the data correctly – ensure that the data you are looking at has been properly defined so that you can be sure to compare it to like data
  2. Compare like with like – be sure that you are comparing 2 sets of data that are measuring the same variable. If you are not then your analysis is going to be out be a long way. If you are applying filters to your data, be sure that you have tested the filters thoroughly otherwise they will skew your data.
  3. If you are able to; ensure that the data you are using has been vetted and checked by a third party. This will help to ensure that there are no underlying complexities that could skew the data tremendously.
  4. Always ensure that you have a backup set of data that you can fall back on. This is specifically relevant when it comes to website and marketing analytics. If there is a second set of data that you can use, which has an acceptable level of variance in it, then this can ratify your data collection and interpretation methods.

Leave a Reply