Big Data and the State of Your Loft

You may have noticed a huge amount of focus on Big Data and its benefits to businesses and their income. I would not disagree with this, but I don’t think I have ever come across a concept that has had so many illustrations, analogies and over engineered stories to explain a fairly simple concept in my entire career that has spanned two different centuries… most vendors agree that the term ‘Big Data’ is not really appropriate anyway!

Therefore I give to you my breathtakingly simple, yet entertaining take on the whole Big Data piece.

So how’s your loft?

“So how’s your loft?”, I ask this not because I hold some desperate need to understand the functional use of your loft space. I ask the questions simply as it relates to the question ‘how’s your data’? Your loft space is very functional, and you can put all sorts of things up there… Data is very useful and you can store lots of different data types.

How does your loft compare now to when you first moved in? Has it gone from a nice neat area to something somewhat indescribable? As the years have gone by, has it now become a complete bomb site with stuff up there that you can’t remember either buying, needing or wanting? perhaps it has a few gems you had forgotten, like that mint condition action figure now worth £100’s of pounds?

The good, the bad and the ugly…

Big Data is simply your loft but in data terms. It will be full of items you will never need, want, or care about, (we won’t go into the ICO’s view on that) but it will almost certainly contain items that you have no idea about, that have a real value to your business or stakeholders.

The key here is how you sort out your data. Unlike your real loft you can’t simply wait for the Easter break, as a business you need an ‘approach’ to sort out the gems from the waste and a plan to exploit them, and of-course you need investment up front to mine the data and understand the return on investment.

Some would say that Big Data is Big simply because of how much data we have hoarded over the years, however in this world of always-live on-demand technologies it will naturally drive data into a place where the volume simply means we struggle to cope with it using traditional BI technologies. Consider the sheer volume of live data handled by the chaps at to understand how you play Candy Crush on your mobile just to learn how best to ‘up-sell’. Nectar card is another example of the ever flowing live data that needs to be stored and mined for business advantage.

What now?

If your loft is full and unmanageable you might have to re-think your filing options (migration to a HADOOP NoSQL platform?). In essence the key to successful Big Data, and the somewhat impossible dream, is a tidy loft where you know where everything is, what it is and it’s value. If your loft is disorganised and you’re struggling to push the loft hatch open (which is usually the case for most of us), then you may need some of the clever tools like to short-cut your approach and discover what’s lurking in your data sets. Other Big Data tools are of course numerously available… (you may need deep pockets though).

So to end on an amusing choice of words, Big Data is certainly a ‘lofty’ goal… Copyright © 2014 Gareth Baxendale

Source by Gareth Baxendale

Leave a Reply

Your email address will not be published. Required fields are marked *