Published: 08 October 2014
The evolution of big data has presented a myriad of challenges and opportunities. It has become increasingly difficult to process and analyse the mountains of structured and unstructured data using traditional technology and logic. The Mckinsey Institute (2011) has estimated a 40% annual growth in global data and with contributors such as social media, government, healthcare, banking and retail, we do not foresee this trend slowing down anytime soon.
Big Data Challenge
The challenge in storing and analysing big data is resulting in missed opportunities to increase revenue, production and customer satisfaction. Businesses rely heavily on data for strategy, competitive advantage and innovation. There exists enormous potential and value in big data if we can leverage the complexities to improve analytics and business model innovations.
Erwitt and Smolan (2012) estimates that the amount of information generated in the less than the 15 minutes it takes to read this article will be approximately 20 petabytes (p. 18). Erwitt et al. (2012) further highlights the magnitude of the big data challenge as more that 99 percent of written words, images, music and data in the world are transmitted digitally (p. 19). If this does not get your attention then maybe knowing that “we are actually just at the beginning of the big data era” (Erwitt et al., 2012, p. 18) will underscore the fact that challenging and exciting times are ahead in the big data era and how we leverage the complexities for innovation and analytics will be a turning point in mankind’s history.
Big Data Revolution
We are indeed on the verge of a data revolution or rather in the midst of it and it is helping to shape how we do business, shop online, use public Wi-Fi spots, online banking, medical treatment and diagnosis and use social media, just to name a few. Amidst all the noise about big data and whether it is a trend or a new cool terminology to describe an evolving and dynamic field, the facts show that we are facing a data crisis as information is generated at an alarming rate and continues to grow each day. The question should be asked, how do we filter what is important and what is not? How do we sift through the mountains of structured and unstructured data to find useful patterns and trends? There is tremendous opportunity to address the challenge that exists with complex big data to improve competitive strategy, customer value, increase growth and revenue.
Big Data Characteristics
The term “Big Data” would speak to the obvious sheer magnitude of data, hence the term big, however big data has four key characteristics:
- Volume: data generated is currently outgrowing traditional storage capacity
- Variety: new data streams are being processed for insight, e.g. social media, email video, blogs etc.
- Velocity: data is generated real time for analytics
- and the not so frequently mentioned Value: identifying valuable data for extraction and analysis
These characteristics also present challenges for organisations to evolve to meet the complexity of big data’s volume, variety, value and velocity (Oracle, 2013, p. 4). Fawcett and Provost (2013) states that “there is convincing evidence that data-driven decision-making and big data technologies substantially improve business performance.” (p 17). Hence, it is crucial that “enterprises have an opportunity to leverage their data to create new revenue streams and generate new businesses.” (Avanade, 2010, p. 3). An area of big data that should also be considered is privacy, as companies collect more data on their customers and begin to paint a picture of their lives; privacy is a major concern as we ask “how much is too much?” Nonetheless, the opportunities far outweigh the challenges and with any new field that grows as rapidly as data science, challenges are unavoidable and they in turn help to drive growth and innovation.
About the author: Raquel Seville [@quelzseville] is a Business Intelligence Professional, BI Evangelist, Founder: exportBI | Co-Founder: eatoutjamaica. To find out more, please visit her about me page.
Avanade. (2010). Global survey: the business impact of big data.
Erwitt, J., & Smolan, R. (2012). The human face of big data. New York: Against All Odds Productions.
Fawcett. T., & Provost, F. (2013). Data science for business: what you need to know about data mining and data-analytic thinking. New York: O'Reilly Media.
Oracle. (2013). An oracle white paper, oracle: big data for the enterprise
[Excerpt from PHD Research Proposal]