People like hype. They're drawn to the next greatest thing, whether that be a new buzzword or a promising trend (or even a scary new trend). For some time now, business and IT have been separated into two camps: one group believing that big data is the revolution that will change everything forever; the other maintaining that big data is basically a meaningless buzzword, and that big data analytics is really nothing more than using traditional analytics on an ever-growing scale.
The industry is ripening with 'big data solutions'. Vendors promise better ways to collect data, store data, and analyze data. There is no doubt that the big guys have, use, and benefit from big data -- just look at the advancements made by big search engines, social media platforms, and the mega-retailers like Amazon and eBay. But when it comes to the ordinary businesses, the mid-size guys that aren't necessarily household names, there is a lot of balking going on when it comes to adopting big data.
The answer: yes, big data is a thing. The data sitting in your current data warehouse may or may not qualify, but even for the small- to mid-size companies, big data is both possible and promising. That's not to say that if you don't have a data initiative already underway that you're hopelessly behind and bound to fail at any moment -- but it does mean that at some point you'll need to do the research to see if big data makes sense for your company. Here are some of the most common arguments against big data adoption, and how you can reasonably respond to these issues in your own company.
Many Are Still Unsure What 'Big Data' Even Is
Let's start with the most basic: do you really have (and do you need) big data? Big data is not characterized by its quantity, but instead by its qualities. You can have enormous stores of transactional data, data on inventory history, etc., but those data sets are generally homogeneous. Big data, on the other hand, is varied. It generally comes from multiple disparate sources, and is usually unstructured -- that is, it isn't formatted into a state that fits neatly into a typical data warehouse.
Say you want to add data from your mobile app with that of your POS transactional data -- this scenario actually doesn't involve big data. Similarly, if you want to take data from your suppliers' systems and analyze that together with your inventory data, you're talking about a pretty typical data warehouse approach to analytics.
The standard criteria for determining whether you're actually dealing with big data are the 3 Vs:
Volume: which just means you're dealing with tons of data
Velocity: which means you're generating and processing the data very quickly, often in close to real time.
Variety: which means the data is coming for multiple sources, internal and external to your own organization--and it's most likely unstructured.
Others are Unconvinced of the ROI for Big Data Initiatives
There are some real horror stories about businesses that took on big data with 'free open-source' products like Hadoop and spent truckloads of money and time without producing a ROI. But if you look carefully, these tales of terror are ancient, at least, on the scale of IT, where a year is a decade and a decade is a century. Big data tools like Hadoop have advanced light years just in the past couple of years.
There are also lots of promising new products that come with far more manageable learning curves and much better results (Spark, for one). While the lack of talent for big data analytics is often cited for poor ROI, the market is remedying this, as well, either by home-growing data scientists (which is definitely possible if you have a team with strong programming and statistical skills) and via numerous degree programs in top colleges around the country and across the globe. Plus, with the right vendors, you can do a lot of data analysis with relatively little staff.
A Lot of Organizations are Concerned About the Privacy Issues Surrounding Big Data
The privacy issues surrounding big data are no different than with any other data store -- you're just guarding a lot more of it.
Finally, there's the argument about privacy. Yes, security and following the industry and government regulations governing consumer privacy are real issues. However, the security and privacy involving big data is no more troublesome than the average consumer database. If you're holding personally identifiable information, credit card and/or social security number information, or other sensitive consumer data, then you're already dealing with the depth, breadth, and scope of privacy issues that you would have with big data. You're just guarding a bigger, richer pile of it.
Aaron Crouch is the Business Intelligence Practice Leader for Aptera. Upon graduating from Purdue University, Aaron immediately began working on Data Warehouses and BI implementations. With 14 years of Business Intelligence experience, Aaron has strategically planned, designed, and developed solutions for multiple Fortune 500 companies both in the United States and abroad. His work spans the fields of retail, education, healthcare, manufacturing, and automotive vertical markets.