Chris Flanagan, Data Impact Manager for Henley Careers, reflects on the phenomenal growth of data analytics and the challenges and opportunities that the rise of Big Data present.

As an information officer starting out in the 1990s, being “good with data”, although considered useful, would have featured fairly low down in the role description; certainly more of a desirable than an essential attribute. The ability to create the occasional pivot table or bar chart about the limit of the expectation. Back then data management was largely about ordered spreadsheets and databases, creating lists and inventories, principally because there actually wasn’t a huge amount of data to analyse. As Bernard Marr, an expert in the intelligent use of data in business, says

“Anything that wasn’t easily organised into rows and columns was simply too difficult to work with and was ignored”

Fast forward 25 years on, with the digital age firmly established the landscape has changed completely: we now have access to vast streams of data that allow us to easily produce ‘actionable insights’. Data analytical skills are in high demand for an ever-increasing number of roles across all industry sectors. My advice to a younger version of me, if I was starting out today looking for sustainable employment well into the future? Get on to a data science or business informatics programme (preferably from our own excellent Informatics Research Centre at the Henley Business School).

It’s the availability and sheer volume of Big Data that creates both huge opportunities and considerable challenges. Apparently, as humans today, every two days we create as much data as we did from the beginning of time until 2000. Read that again: data produced every two days greater than data produced in first 200,000 years of human existence…

How can this be?

Marr argues that in almost everything we do we are leaving a digital footprint, or generating a data trail, mainly through our use of always connected GPS-equipped smartphones: how many times a day are we more than a few feet away from our phones, perhaps for some of us the thing that wakes us up in the morning and the first thing we check before getting out of bed to the last thing we put down in the evening before switching off? It’s not just the data that we generate: the volume of machine-generated data is rapidly growing too, for example smart home devices such as Hive (communicating and sharing data) or traffic sensors gathering and transmitting data.

Advances in storage and analytics mean that we are now able to capture, store and work with many, many different types of “data”,  anything from databases to photos, videos, sound recordings, written text and sensor data. And it’s the machines that will help us to make sense of the vastness and ‘dis’-order of the non-traditional data forms – as Marr says:

“to make sense of all of this messy data, Big Data projects often use cutting-edge analytics involving artificial intelligence and machine learning. By teaching computers to identify what this data represents– through image recognition or natural language processing, for example – they can learn to spot patterns much more quickly and reliably than humans.”*

Big data is already bringing huge benefits to business in terms of reducing costs, providing faster and better decision making, and by using analytics to gauge customer needs the ability to bring more focussed and streamlined products and services to market – giving customers what they want before they ask.

So do we all need to upskill, to enrol for an Advanced Level course in MS Excel to stay in the game?

Well, perhaps not in terms of preparing and tidying data as the machines will do much of this heavy lifting to allow for better predictive analysis to help businesses overcome some of these challenges of processing so much data. But in terms of being able to review and interpret data, then yes, these skills are becoming ever more the expectation, the norm. Fortunately, in a relatively short space of time tools for non-coders will be available to help gather insights, to read the signals and not be distracted by the noise that all this new data is creating.

Plenty of opportunities then, but what about the concerns? Well, unless you’ve been living under a rock for last two years, you’ll be all too aware of changing data regulations aka GDPR. The principles GDPR seeks to embed are very relevant to concerns around Big Data and can actually help us through some of the challenges:

• Data privacy – the balance between the amount of personal data we divulge for the convenience and benefits that our favourite apps and services offer.
• Data security – Even if we decide we are happy for someone to have our data for a particular purpose, can we trust them to keep it safe?
• Data discrimination – this is something my colleague, Dan Kiernan, spoke about during his session on Fintech at last year’s World of Work conference with reference to the insurance industry where it is now possible to price risk individually rather than by pooling risk. This might be better for some of us but is it fair to discriminate in this way, with certain people, often the most vulnerable and those with the least resources, being excluded from being able to obtain insurance or other key services?

So as great as the opportunities are, these are some of the challenges that we must all seek to address. For all the technological advances taking place around us it’s good old-fashioned values of governance and ethics which need to be at the core of how we utilise the vast benefits that Big Data might provide.

Interested in Big Data? Join us at our World of Work conference on 13 September as we enlighten challenges such as this, and the dark side of the future of work. 

*https://www.forbes.com/sites/bernardmarr/2016/03/15/17-predictions-about-the-future-of-big-data-everyone-should-read/#8d17d871a32d