We are living in the golden age of big data.
1 Jun 2015 by Evoluted New Media
We are, I hardly need point out, living in a golden age of big data. It is completely changing the way most science is done; indeed some science can only exist because of our relatively recent ability to handle enormous data sets.
This is a marvellous, exciting and vital thing for science – it has, however, given me a rather unusual problem.I find myself more and more consumed by the superlatives and analogies used to describe big data.
I seem innately programmed to hoard trivia; exactly the kind of trivia the sharp-suited marketeers involved smugly evangelise when they communicate about big data.
If you burned all the data created in one day onto DVDs you could stack them on top of each other and reach the moon twice…Today’s data centres occupy an area of land equal in size to almost 6,000 football fields…The number of bits of information is thought to have exceeded the number of stars in the physical universe in 2007.
Lovely trivia tidbits, each and every one. And there are just so many of them. Frankly it’s a little overwhelming. In an irony that seems quite ridiculous, I have even considered setting up a database in order to capture all of the facts contained within the various infographics I get bombarded with on an almost daily basis. Honestly, I think I need a big data approach to deal with the ever increasing amount of big data descriptions.
So, if my (admittedly average) mind can barely keep up with the descriptive terms of all this data, how can it be expected to actually handle the data? How could anyone’s?
Of course it couldn't – the key to unlocking the information held in these enormous data sets wasn't just the technology to store and process them, nor the methodology to collate them in the first place; we had to learn how to extract meaning and display it in a digestible way. We had to turn big data back into small data.
And there is no better example of what an enormous challenge this is than the Large Hadron Collider. The experiments attached to this most famous of proton lobbers generate an almost obscene amount of data. And, never to be outdone, they have a mightily impressive set of statistics – some of which you can see in this month’s number cruncher.
The task of processing the spoils of one of these experiments, ATLAS, is explored in our big data feature – and the trick, like so many things, is effective, organised partnerships.
As if the front-line experiments at the LHC weren’t collaborative enough, in the handling of its data we see an even wider net cast. It seems to me that collaboration billows from big data generators like the LHC as if ripples in a pond…such a wonderful thought it has almost distracted me from the latest bit of trivia.