Big Data Analytics – 5 ideas to consider

As someone who drives complex ideas through and around dozens of roadblocks, I am often reminded that one of the best ways to help people is simply to highlight the right resources that will eventually transform into knowledge and wisdom.

In one of the most complex areas of my endeavors is the current ‘seizure’ mentality that is stopping the right technology and trends at the appropriate levels. This is especially true of the Big Data conversation that is challenging all levels of business.

Ironically the small and mid-sized business teams are having an easy time gaining traction in understanding the issues that affect their business. The small and medium businesses bring another problem: not having enough Big Data to play with and the budget to bring the expertise to make the initial change.

So the key to think about:

Enterprise Player = focus on tactical and economy of scale.

Small and Mid-sized Business = focus on wisdom of larger data providers.

If you are in an organization that is currently considering a Big Data initiative, please  take a moment to reach this article by John Weathington over at TechRepublic = To get Big Data buy-in, IT should let go of proof of concept …

Once you have had a chance to read it, come back and explore some of these other Big Data articles with the mindset of initiating the process with the right team and executive support in place. The simple reality is that Big Data isn’t an IT issue; it is a game changing opportunity for executive leadership and owners who choose to be nimble.


Dynamic data analysis the future of big data: Brainmates – ZDNet

Big data isn’t all about beefing up marketing strategies for organisations, according to founder of the product management …

“As a predictive tool, big data can deliver better products, but to effectively achieve this, product managers should use big data to test the questions that have previously been impossible to answer until after the product has gone to market,” he said. “Too often, we are led by what we think we can do and are trapped on those rails, whereas if we imagined the impossible, pretty soon we will find it has already been done elsewhere.”

Can Big Data make government cheaper? – ZDNet (blog)

The movie “Moneyball” celebrated a “Big Data” approach to maximizing returns from investments. Can Big Data do the same for government?…

“Today’s young people – hammered by the Great Recession, the loss of middle class jobs and gridlock in Washington – may take a more pragmatic approach. Stress – like the Great Depression and WWII – seems to focus Americans on solutions rather than ideology.”

Putting a Dollar Value on Big Data Insights – Wired

The Big Data phenomenon produces some mind-boggling statistics, such as the fact that the volume of global data produced doubles every two years. At the University of California at Berkeley, researchers …

” To cite just one example, retail giant Walmart was able to use Big Data analysis to drive a 10%-15% increase in completed online sales for $1 billion in incremental revenue, which is a well-planned ROI using data.”

5 Big Data Projects That Could Impact Your Life – Mashable

GCN argues that the first “big data” software was IBM’s DB2, a database management system it released back in 1983. What’s new is the ability to compile and process everything at such large scale …

“It’s a swanky term for a not-so-sexy concept: the idea that mass amounts of information can be analyzed to find hidden patterns, buried beneath terabytes of numbers, in Facebook posts,Google searches and Amazon purchases. These patterns can predict social trends and, in some cases, reengineer the way we live.”


Enhanced by Zemanta

Big Data Brokers and Open Data Sources

The current state of the world is that we almost every action and fact we take are tracked and recorded; ranging from mobile phone usage, driving behaviors, social networks, and our shopping transactions.

Due to the nature of data and the fact that it exist in an invisible layer that most of us don’t comprehend, the results is that many of us don’t realize the ‘big business’ of our data.

Without our knowledge and comprehension there are hundreds of big data brokers who sell and barter our information for a wild variety of uses. These organization not only make massive amounts of profit by selling information we created, but they also make tremendous impacts to the bottom line revenue of businesses who understand how to use big data.

As I go through this article  (which has turned into a book) I’m going to give you a thought as to the scope of this business and why you should pay great attention to it both personally and professionally, along with giving you pointers to some of the available sources of data (ranging from open, governmental, local, industry, etc.)

We’ll examine some critical areas around big data

  • Why to use big data and open data sources regardless of how big or small your business is
  • Common areas to browse through and familiarize yourself with
  • Understanding what trends are affecting your future

Getting Started with the big question…

What is my data worth?

Part of answering the bigger questions around big data brokers and open data sources is the need to understand what your data is worth and that for every person, group, and organization there is an enormous scale of data being created.

Consider this, for every person: They interact with 100+ other individuals, 100+ local businesses, and 1000+ global businesses.
Each interaction creates a data point. Each data point has value. 

This federal action helps illustrate the scope of value: Oobama-executive-order-open-datan May 9th President Obama issued an Executive Order directing historic steps to make government data more accessible for economic growth and innovation. The order declares that the information is a valuable resource that needs to be used as a strategic asset for the nation. President Obama called for an Open Data Policy, requiring machine readable formats of data for government records that can drive long-term economic growth.

The origination of the order comes from growing concern including personal privacy, regulation, industry development, and competitive positioning of ‘who owns your data’

If the President of the United States is issuing an Executive Order regarding government data and considers it a national asset, chances are you should really be paying attention to the types of data you create and interact with.

By the People, for the People.

In an on-going series of efforts going back a decade, our government has a series of sites providing what I would refer to as an elementary view into some of the data we create. On a rating of 1 to 10 and of usability to the common person or business this data is a 1, it has questionable freshness, unidentified quality, and lack luster categorization.

The idea is there, but what the government is providing is really ‘only data’ –  It doesn’t come in fancy spreadsheet or colorful visualization that make you understand the significance of the data. This prevents the common professional and business owner from making decisions regarding on-going trends ranging from local, regional, and global perspectives.

Examples of basic government data

We also have Open Data

Open data is publicly accessible information that has been established by public interest groups, government, corporate entities, and expert practitioners.

The general idea behind Open Data is that even big corporations need to release some ideas about the information they have access to so that innovative entrepreneurs and community minded leaders can build interesting things with that data.

Example sources of Open Data


Big Data Brokers

Take all of your data, along with governmental and open data, then buy some more!

Big Data Brokers collect all sorts of data ranging from transaction and purchase behavior, consumer interest information, and usage patterns. The basic concept of a big data broker is that they collect merchant details at the point of sale, social data from your relationship network, and demographic identifiers from millions of different databases that have been collected under a few hundred organizations that fall into the category of being a ‘data broker.’

They sell basic pieces of data such as names, contact information, and additional demographics, like age, race, occupation and education level. There are lesser known data areas that include everything from how fast you drive your car (insurance reports, traffic records) to what your entire employment history looks like.

You also have big data brokers that sell information about products (such as Amazon store data), places (Google Maps), events, vendors, manufacturers, employee names, and stocks (Nasdaq, Dow Jones), etc.

The list is truly endless.

On the downside of big data brokers is the fact that there is someone willing to track everything and anything related to you. I’ve had the pleasure of speaking about the topic of privacy and invasive trends that affect us on a multitude of levels, but in most cases the ‘big data’ being collected about you is years ahead of legislation and regulation.


Think about everything you type into Google, Facebook, or your Mobile Phone.
Take every e-mail, text, conversation, or transaction and think about where all of the data goes.
Imagine the past five years of your  data collected on a computer and offered up for sale for $1

While the privacy issues are enormous, the beneficial areas are enormous as well.

Overlapping Points to Consider

We can think of the overlapping open, governmental, and big data segments in several different viewpoints that range across three primary categories: the individual, the business, the community.

Each of these categories have signals of data that reach far beyond the boundaries of the specific element. The visual below helps demonstrate how waves of data interact at dozens of points from the three categories.


  • For the individual big data has the risk of invasion of privacy, with the benefit of streamlined performance and increasing the quality of life. The same data about your home electric consumption can be used against you to market goods and services, or used for you to help reduce usage and be environmentally aware.
  • For the business side of data we run the risk of being the instigators of breaching personal privacy, while we have the benefits of improving income and employee/consumer lifestyle. We need to be aware of local and global differences in ethics, morality, and legality as we engage our businesses as both consumers and providers of competitive marketplaces.
  • For the community portion of big data we run a balancing act of considering privacy of the individual, the trends of our community, and the financial interactions of our community businesses. We also need to consider the asset of data as a community, as the risk and benefit for both individuals and businesses are almost entirely controlled by representation as a community (in the form of a real world community, association, employer, etc.)

What is all of this data used for?

That is a pretty big question. The most typical use is one of two things:

  • Option 1: attempting to collect enough data to reach a critical mass that is easy to sell. Most social networks (Facebook, etc) and search engines (Google, etc) need to collect billions of data points before there is enough quantity to sell the data over and over again in some form of profitable model.
  • Option 2: they are attempting to find areas of overlapping data that have a point of interaction that identifies a recurring situation or life-event trigger such as buying a new smartphone, getting married, or ordering a movie on demand.

How do I use this in business?

If we imagine both option 1 and option 2 happening at the same time, we could look at an example of several business entities sharing a common community. This could be a group of industry businesses, a few stores working together at the local mall, or perhaps international trade partners shipping into the same city.


When a new individual interacts with the community or any one of the three businesses, a data exchange is possible.

In the real world this could be thought of as a friendly referral (it can have darker tones as well.)

When the new individual is introduced, the data points they bring with them have dozens of interaction points with the three vendors.

big-data-brokers-for sale

The are important things to realize about the above chart.

  • When an individual first engages it is typically with a community layer.
  • The individual has data that reaches into the core of one business (perhaps a transaction) and has only secondary data touching the other two.
  • All three businesses collect different data points on the same individual.
  • The individual may be aware of one, two, three or none of the businesses.

How should I use Big Data?

(Disclaimer: this is entirely my opinion.)

Most big data is used to sell you stuff. This data exists about a tremendous number of things and you can use it to sell all sorts of ‘stuff.’
Making money and driving a strong economy can be a great reason to use big data.

As a serial entrepreneur and innovation consultant, I prefer to give people a big idea to go with big data. 

You can sell the data…You can buy the data… Or…

Change the World with Big Data.

You can build things with data.

You can inspire people with data.

You can change lives with data.

If you want to so do something big-  think about using big data to help your community, people who do not have access to the data, or your family. Look at your network of friends, family, and co-workers and consider the insights that are hidden in your data.

  • Take a look a local charities and think about how your data aligns with something that makes you feel good.
  • Look at local employment rates and use the data to make jobs for people who need them.
  • Use data to reduce city spending, improve the education system, or help solve homelessness.
  • Spend some time looking through the open data and government data above.
  • Think about ways you can visualize data to help people understand.

I hope this has given you some insight to
the world of Big Data.
Go and do something useful with it!


Some interesting context to specific Big Data issues

Health Data

Forbes: What the Government’s Big Data Dump Tells Us About the Hospital Market –  I believe in the power of markets to create efficiency and innovation, so I will go there. To unleash the power of entrepreneurs to create transparency and choice in the hospital services market. Fortunately this starting to happen. Companies like CastLightHealthInReach, and Pokitdok are good examples.

Data Value

AdAge: FTC Sting Operation Results in Warnings to 10 Data Brokers – The FTC “test-shopped” for data from the companies and determined that they may be in violation of the FCRA, according to the missives, dated last week. The firms in the crosshairs are ConsumerBase, Brokers Data, US Data Corporation,, 4Nannies, U.S. Information Search, People Search Now, Case Breakers and USA People Search. The tenth company was not named by the FTC.

 Privacy Data

IT World: Web Trackers are Out of Control – The amount of data compiled about me was jarring. BlueKai and its tracking partners (DataLogix, Lotame, and others) had aggregated some 471 separate pieces of data. On the other hand, BlueKai correctly determined that I have two teenage kids, I’ve lived in the same place for more than 7 years, carry a mortgage, and like sci-fi movies, travel, and politics.

Facebook – Big Data – Our Privacy

Last week I had the opportunity to speak at the 2012 Privacy Identity and Innovation conference (PII 2012)  in Seattle.

My on-going observation of privacy and identity actually revolves around the connection point of big data and how corporate entities are trying to monetize it.

We need to realize the concepts behind how businesses use our personal data affects the very way we make choices as individuals.

Google doesn’t just connect us with content.
It affects our perception of content

Facebook doesn’t just connect us with friends.
It affects our perception of friends.

Big Data isn’t just about our purchase decisions or where we are, it has to do with every data point that we can create or interact with. It could be anything from the time of day we decide to e-mail our loved ones or the list of people we share with online. Big data also interacts with the information we don’t always think about or fully comprehend: our location, our mood, our activity, our habits. Read more

PII 2012 – Brand Reputation: The Role of Privacy in Communication

I’m looking forward to moderating a panel at the Privacy Innovation Identity
conference( PII 2012 ) next week.

Join our panel on Tuesday the 15th to hear a great conversation and engage the experts.

Brand Reputation: The Role of Privacy in Communications

•    Moderator: Barry Hurd (Linkedin), Managing Director, Epiphany Metrics @BarryHurd
•    Mary Ludloff (Linkedin), VP, Marketing, PatternBuilders @mludloff
•    Leigh Nakanishi (Linkedin), Privacy and Security Strategist, Edelman @LeighNakanishi
•    Mike Whitmore (Linkedin), President, Fresh Consulting @MikeWhitmore Read more