Compared to all the data we could be using to understand and act on issues and opportunities, many of our conventional data sets are woefully limited. This is often a situation of our own making. We collect only very precise types and quantities of data, which we then put into rigidly defined databases or other systems of record. These databases give us answers to questions like “What was customer X’s last purchase?” or “What is employee Y’s date of birth?” Even results from analytic processing systems processing complete datasets like these are limited by the data they process and the way they process it.
Big Data Uses and Examples
Big data goes beyond these restrictions by allowing any type of data to be used from any source. Instead of purely employee-entered data, for example, big data extends to customer or constituent comments on social media, weather forecasts, third-party economic forecasts, machine sensor outputs and any other data source that has a bearing on the questions being asked or the issues to be resolved. Analyzing big data to reveal patterns, trends and likely outcomes can then take organizations far beyond the limits of “normal” data processing, as these examples show:
- Situation analysis. Big data analysis that includes unstructured text comments from staff and patients, as well as healthcare service and financial records, can show how well a hospital is doing its job.
- Business objectives. Historical and projected vehicle sales data by type of vehicle, urban development projects and forecasts, and surveys on user preferences for different modes of transport are possible big data inputs to help plan road infrastructure for the next decade.
- Strategy development. Different models of power generation (fossil fuel, renewable, nuclear, for example) can be investigated before a final strategy is selected for best meeting future power needs. Big data inputs can help model carbon emissions, costs, impacts on residential and industrial sectors and even public and enterprise reactions.
- Crisis management and public affairs staff can assess effectiveness in dealing with situations or communicating decisions or policies, by analyzing sentiment and feedback from social media as well as “hard data” like time and costs to contain or communicate.
Getting to Grips with Big Data Processing
Big data is a collection of data too large for traditional business applications to store, process or manage. Harnessing the power of big data to yield insights and offer predictions for the future requires other tools to handle the “four Vs”: volume, velocity, variety and value.
- Volume. Although often made up of many smaller data sets, big data can vary from terabytes to petabytes in volume.
- Velocity. Big data must be captured and analyzed fast enough for the results to be up to date. Coming up with yesterday’s insights only today may be useless in fast-moving markets and environments.
- Variety. Big data can take any form, including structured data like SQL database records, unstructured data like social media content, or a mix of both.
- Value. Insights and predictions must be useful and offer a good return on the investment of resources needed to obtain them.
The most useful results from big data do not arrive simply by “turning a handle.” Many organizations are still identifying the data relevant to their goals or improvements desired. People working with big data need not only the right tools, but also knowledge of the subject area concerned, and the ability to think critically. With that said, things are moving in the right direction and recent developments in cloud computing services now make it easier and more cost-effective to access resources for big data management, analysis and insights.