The Devil is in the Details

Big data, that "large volume of data - both structured and unstructured - that inundates a business on a day-to-day basis" is a hot topic these days.

Even though the "information explosion" actually started 70 years ago, an increasing volume of online content has created a sudden rush for businesses looking to big data as the next money maker. In fact, the big data industry is expected to grow from $10.2 billion (US dollars) in 2013 to $54.3 billion by 2017. 

If used effectively, big data can lead businesses to "more accurate analyses of vital information, leading ultimately to greater operational efficiencies, cost reductions, reduced risk, speedier innovations, and increased and new revenue." That said, big data's importance is in user data and this makes "users more central to product development in the form of detailed data about their behaviour."  

But with all of its potential good, big data also has a dark side: The bigger the data, the bigger the consequences from potential privacy breaches. One need only think of the Ashley Madison or WikiLeaks cases to consider the outfall of damaged lives in this situation.  

To avert these types of potential privacy concerns, those collecting data must be diligent about the type of data collected and reasons for its collection and use.  

And these privacy concerns are directly related to data security.  

Data theft is a rampant and growing area of crime and according to Bernard Marr, five of the six "most damaging data thefts of all time" occurred within the past two years. It is not an overstatement to say that the bigger the volume of data, the bigger the potential security breach.  

In addition to privacy and security, cost is also a consideration in big data. Data collection, aggregation, storage, analysis, and reporting are not free. Companies like Google, eBay, LinkedIn and Facebook may have unlimited budgets for their data, but smaller businesses are at a disadvantage in this area. That is why it is important to collect only the data that is required for the intended purposes.

In other words, starting with a strategy is key.  

By having a strategy, organizations limit their data collection to only the information that they need. This results in lower overall costs. As well, less data means that less time is required for data analysis. 

Big Data to the Rescue

How easy is it to analyze multiple datasets and decipher consumer behaviour? It turns out that even Google can get it wrong.

Google's "Flu Trends" project set out to produce accurate maps of flu outbreaks based on searches made by Google users, but the map diverged from reality as time went on. It turned out that Google's algorithms weren't accurate enough to pick up anomalies such as the 2009 H1N1 pandemic. This reduced the data's value.  

In addition to bad analytics, bad data can also produce skewed or irrelevant results. The problem for businesses, in this instance, is that they will lag behind their competitors if their data or analytics are incorrect.  

Consider the difference between Uber and the taxi industry. Uber's use of good data has propelled it to become a big data company (rumoured at 800% annual growth) while the taxi industry's bad analytics, or perhaps lack of attention to data, has left it floundering offline.  

In addition to bad analytics, organizations have the potential of producing bad data or not collecting the right data in the first place. For instance, collecting data just for data's sake can lead to a collection of bad data that has no value in decision making. 

As a way to stay ahead of the competition, big data trends include smart devices capable of voice recognition which feature collects and downloads data to the vendor (some people feel that this feature is akin to "Big Brother" watching).  

On the other end of the spectrum, the CIA and NSA proclaimed that "we kill people based on metadata. Potentially, big data poses serious threats to citizens. And in recent news, big data is proposed as a way to monitor social media for signs of mental illness. 

Another big trend in big data is job opportunities. The market is lacking data experts. For example, in 2013, Ritchie Bros. Auctioneers could not find a single student to hire for their data expert position and, instead, hired a professor who taught business intelligence at the University of British Columbia.  

While big data-related professional services are expected to grow at a compound annual growth of 23 percent through 2020, there is a related staff shortage that extends from data scientists to data architects and experts in data management.  

As the Internet heats up and big data gets stronger, Rossi  says that "the song remains the same: everyone's a data analyst, and there's never been a more exciting job." And as the future nears, time will tell if Rossi is on point about the excitement around big data or if big data turns out to be a big bust.  Either way, there is no ignoring the big data trends that are now shaping our lives. Organizations that ignore the big data trends do so at their own peril.