Finding Insight in a Digital Sea of Information

For roughly a decade, businesses and governments have been collecting digital information about the world’s citizens; not just names and numbers, but loads of data, Big Data.

The IDC (International Data Corporation) predicted that the Big Data industry will experience a 26.4% compound annual growth rate, reaching $41.5 billion by 2018. What this means is that more and more companies and organizations are paying top dollar to media agencies to be deluged in never-ending monsoons of big data, and lots of money is being spent figuring out how to harness its power. Chris Mattmann, from the big-data initiative at the NASA Jet Propulsion Lab (JPL) in Pasadena, Ca says, "NASA in total is probably managing several hundred petabytes, approaching an exabyte..."

An exabyte. That’s a lot of data. It’s one billion gigabytes, which numerically, looks like this: 1,000,000,000,000,000,000.

Big Data For Big Firms

These enormous data sets proliferate so quickly that conventional database technology is not capable of storing or analyzing them, and the thing is, an organization doesn’t need to be focused on deep space to be awash in a sea of big data. Chris Riquier, the Asia Pacific CEO of TNS (Taylor Nelson Sofres), says big data has affected market research. “Market research,” says Riquier, “was founded on surveying and polling.”

He describes how in the past, surveys and their processing would take weeks, often rendering the data analyzed obsolete to businesses, and goes on to report that by aggregating social media data, search data and other forms of big data, we now have the opportunity to “rethink how research is done.” As a result of big data, says Riquier, our “ability to react to the market and make decisions has changed drastically.”

In today’s digital world, big data finds relevance across industry, government, science, public health and academia, however gaining insight from all this information has been challenging for most, until recently.

According to an article published in 2014 by Harvard Magazine, entitled ‘Why “Big Data” is a Big Deal’, improved statistical and computational methods that involve linking datasets, visualizing data, and creating “big algorithms,” are the key innovations that will enable us to wrestle all this data to the ground.

Taking cues from physicists and astronomers, who have long since worked with huge datasets, data scientists and social scientists are combining the quantitative with the qualitative to gain insight from big data. Big data analysis is actually creating a new field, with Harvard’s School of Engineering and Applied Sciences working toward offering a master’s in Data Science.

In their book “Big Data - A Revolution That Will Transform How We Live, Work, and Think,” Viktor Mayer-Schonberger and Kenneth Cukier talk about how companies are changing the ways they make decisions, based on their analysis of big data.

For example, Google recently used its stored searches to accurately predict flu outbreaks across the US, faster than the CDC (Centers for Disease Control) could, which represents a significant advance in medicine.

According to an article in the Wall Street Journal, Canadian Bank is using Hadoop, an open source software framework created by Apache, that stores and processes big data, to roll out a program that enables the identification of money laundering and fraud.

Big Data For Everyday Managers

Big data has some lofty goals. Incredible minds at Harvard, NASA, Google and Apache are all taking advantage of the power big data analysis brings to bear on advancing technology and the world at-large, but as I mentioned earlier, it’s not just rocket scientists who can benefit from big data. Let’s take a look at a couple of the aspects of big data collection and management that all businesses can use.

One major use of big data in marketing is for search engine optimization (SEO). Business owners can use tools offered by search engines like Google and Microsoft’s Bing, combined with various social media data to glean helpful insight on building internet marketing campaigns.

Joe Hall, who focuses on the intersection of SEO and big data through his consulting firm Hall Analysis, says there are two ways to approach big data for SEO. “The first is working with giant data sets that are directly relevant to a specific business,” he says. “Many times this means working with a large brand or enterprise that attracts lots of data.”

Hall cites an example of a client with approximately 16 million backlinks, or links pointing from other websites to the client’s website. These links are an important ranking factor for Google and other search engines. “At that level it changes the rules for backlink analysis,” Hall says, explaining that the size of the data set requires robust processing power for tasks like pattern analysis.

The second way Hall says companies can leverage big data for SEO is to develop a more defined situational awareness. This might mean running correlation studies to have a better understanding of the ranking factors, or user behaviour analysis such as organic click-through rates within ranking results pages.

Both of these types of analysis help SEO professionals develop a "bigger picture," and both require large data analysis to achieve clarity. If you believe big data might be relevant to your online marketing efforts, you might add it to your list of questions the next time you go to hire an SEO firm.

Another way big data can be used in business is to gain customer loyalty. Let’s say I’m the founder of a start-up, and after my first year in business, I take an extremely well-earned vacation where I’ll sit on the beach and read, real books, nothing on screens. At the airport, the ticket agent informs me that my suitcase full of books is five pounds over the limit and I’ll need to pay extra for it. 

What the ticket agent doesn’t know, is that as the CEO and founder of a successful startup, I’m about to become someone who’ll log thousands of hours of flight time and that half of my staff will too. If this airline used big data to aggregate info from sources like credit card companies, social media feeds, blogs, and hotels, they may have waived the fee for my barely too heavy suitcase and gained the loyalty of an excellent customer.

Whether you’re at the helm of a huge corporation, a doctor doing cancer research, or the owner of a one-man online store, big data will make a difference in your bottom line.

We are now able to harness big data to gain amazing insights into our work. As we embrace the power of data-driven decision-making, we move into an age of limitless connection, that will inevitably alter the way we think about the world for all time.

Today’s generation of children are born into the digital age.

Tomorrow’s generation will be born into the age of big data.

This column was originally published on Forbes. Josh Steimle is the CEO of MWI, a digital marketing agency with offices in the U.S. and Hong Kong.