Big data
Big data[1][2] is a
collection of data sets so large
and complex that it becomes difficult to process using on-hand database
management tools or traditional data processing applications. The challenges
include capture, curation, storage,[3] search,
sharing, transfer, analysis,[4] and
visualization. The trend to larger data sets is due to the additional
information derivable from analysis of a single large set of related data, as
compared to separate smaller sets with the same total amount of data, allowing
correlations to be found to "spot business trends, determine quality of
research, prevent diseases, link
legal citations, combat crime, and determine real-time roadway
traffic conditions."[5][6][7]
As of
2012, limits
on the size of data sets that are feasible to process in a reasonable amount of
time were on the order of exabytes of data.[8][9] Scientists
regularly encounter limitations due to large data sets in many areas, including
meteorology, genomics,[10] connectomics, complex
physics simulations,[11] and
biological and environmental research.[12] The
limitations also affect Internet search, finance and business
informatics. Data sets grow in size in part because they are
increasingly being gathered by ubiquitous information-sensing mobile devices,
aerial sensory technologies (remote sensing),
software logs, cameras, microphones, radio-frequency identification readers,
and wireless sensor networks.[13][14] The
world's technological per-capita capacity to store information has roughly
doubled every 40 months since the 1980s;[15] as of
2012, every
day 2.5 quintillion (2.5×1018)
bytes of data were created.[16] The
challenge for large enterprises is determining who should own big data
initiatives that straddle the entire organization.[17]
Big data
is difficult to work with using most relational database management systems and
desktop statistics and visualization packages, requiring instead
"massively parallel software running on tens, hundreds, or even thousands
of servers".[18] What is
considered "big data" varies depending on the capabilities of the
organization managing the set, and on the capabilities of the applications that
are traditionally used to process and analyze the data set in its domain.
"For some organizations, facing hundreds of gigabytes of data for the
first time may trigger a need to reconsider data management options. For
others, it may take tens or hundreds of terabytes before data size becomes a
significant consideration."[19].
Definition
Big data usually includes data sets with sizes beyond the ability of commonly used software tools to capture, curate, manage, and process the data within a tolerable elapsed time. Big data sizes are a constantly moving target, as of 2012 ranging from a few dozen terabytes to many petabytes of data in a single data set. With this difficulty, new platforms of "big data" tools are being developed to handle various aspects of large quantities of data.In a 2001 research report[20] and related lectures, META Group (now Gartner) analyst Doug Laney defined data growth challenges and opportunities as being three-dimensional, i.e. increasing volume (amount of data), velocity (speed of data in and out), and variety (range of data types and sources). Gartner, and now much of the industry, continue to use this "3Vs" model for describing big data.[21] In 2012, Gartner updated its definition as follows: "Big data are high volume, high velocity, and/or high variety information assets that require new forms of processing to enable enhanced decision making, insight discovery and process optimization."[22
Examples
Examples include Big Science, web logs, RFID, sensor networks, social networks, social data (due to the social data revolution), Internet text and documents, Internet search indexing, call detail records, astronomy, atmospheric science, genomics, biogeochemical, biological, and other complex and often interdisciplinary scientific research, military surveillance, forecasting drive times for new home buyers, medical records, photography archives, video archives, and large-scale e-commerce.Big science
The Large Hadron Collider experiments represent about 150 million sensors delivering data 40 million times per second. There are nearly 600 million collisions per second. After filtering and refraining from recording more than 99.999% of these streams, there are 100 collisions of interest per second.[23][24][25]- As a result, only working with less than 0.001% of the sensor stream data, the data flow from all four LHC experiments represents 25 petabytes annual rate before replication (as of 2012). This becomes nearly 200 petabytes after replication.
- If all sensor data were to be recorded in LHC, the data flow would be extremely hard to work with. The data flow would exceed 150 million petabytes annual rate, or nearly 500 exabytes per day, before replication. To put the number in perspective, this is equivalent to 500 quintillion (5×1020) bytes per day, almost 200 times higher than all the other sources combined in the world.
Science and research
- When the Sloan Digital Sky Survey (SDSS) began collecting astronomical data in 2000, it amassed more in its first few weeks than all data collected in the history of astronomy. Continuing at a rate of about 200 GB per night, SDSS has amassed more than 140 terabytes of information. When the Large Synoptic Survey Telescope, successor to SDSS, comes online in 2016 it is anticipated to acquire that amount of data every five days.[5]
- Decoding the human genome originally took 10 years to process; now it can be achieved in one week.[5]
- Computational social science — Tobias Preis et al. used Google Trends data to demonstrate that Internet users from countries with a higher per capita gross domestic product (GDP) are more likely to search for information about the future than information about the past. The findings suggest there may be a link between online behaviour and real-world economic indicators.[26][27][28] The authors of the study examined Google queries logs made by Internet users in 45 different countries in 2010 and calculated the ratio of the volume of searches for the coming year (‘2011’) to the volume of searches for the previous year (‘2009’), which they call the ‘future orientation index’.[29] They compared the future orientation index to the per capita GDP of each country and found a strong tendency for countries in which Google users enquire more about the future to exhibit a higher GDP. The results hint that there may potentially be a relationship between the economic success of a country and the information-seeking behavior of its citizens captured in big data.
Government
- In 2012, the Obama administration announced the Big Data Research and Development Initiative, which explored how big data could be used to address important problems facing the government.[30] The initiative was composed of 84 different big data programs spread across six departments.[31]
- Big data analysis played a large role in Barack Obama's successful 2012 re-election campaign.[32]
- The United States Federal Government owns six of the ten most powerful supercomputers in the world.[33]
- The NASA Center for Climate Simulation (NCCS) stores 32 petabytes of climate observations and simulations on the Discover supercomputing cluster.[34]
- The Utah Data Center is a data center currently being constructed by the United States National Security Agency. When finished, the facility will be able to handle yottabytes information collected by the NSA over the Internet.[35][36]
Private sector
- Amazon.com handles millions of back-end operations every day, as well as queries from more than half a million third-party sellers. The core technology that keeps Amazon running is Linux-based and as of 2005 they had the world’s three largest Linux databases, with capacities of 7.8 TB, 18.5 TB, and 24.7 TB.[37]
- Walmart handles more than 1 million customer transactions every hour, which is imported into databases estimated to contain more than 2.5 petabytes (2560 terabytes) of data – the equivalent of 167 times the information contained in all the books in the US Library of Congress.[5]
- Facebook handles 50 billion photos from its user base.
- FICO Falcon Credit Card Fraud Detection System protects 2.1 billion active accounts world-wide.[38]
- The volume of business data worldwide, across all companies, doubles every 1.2 years, according to estimates.[39]
- Infosys has also launched the BigDataEdge to analyse the Big data.[40][41]
- Windermere Real Estate uses anonymous GPS signals from nearly 100 million drivers to help new home buyers determine their typical drive times to and from work throughout various times of the day [42]
International development
Following decades of work in the area of the effective usage of information and communication technologies for development (or ICT4D), it has been suggested that Big Data can make important contributions to international development.[43][44] On the one hand, the advent of Big Data delivers the cost-effective prospect to improve decision-making in critical development areas such as health care, employment, economic productivity, crime and security, and natural disaster and resource management.[45] On the other hand, all the well-known concerns of the Big Data debate, such as privacy, interoperability challenges, and the almighty power of imperfect algorithms, are aggravated in developing countries by long-standing development challenges like lacking technological infrastructure and economic and human resource scarcity. "This has the potential to result in a new kind of digital divide: a divide in data-based intelligence to inform decision-making."[45]
Technologies
DARPA’s
Topological Data Analysis program seeks the fundamental structure of massive
data sets.
Big data
requires exceptional technologies to efficiently process large quantities of
data within tolerable elapsed times. A 2011 McKinsey report[49] suggests
suitable technologies include A/B testing, association rule learning, classification, cluster analysis, crowdsourcing, data
fusion and integration, ensemble
learning, genetic
algorithms, machine learning, natural language processing, neural networks, pattern
recognition, anomaly detection, predictive
modelling, regression, sentiment
analysis, signal processing, supervised and unsupervised
learning, simulation, time
series analysis and visualisation. Multidimensional big data can
also be represented as tensors, which can be more efficiently
handled by tensor-based computation,[50] such as multilinear subspace learning.[51]
Additional technologies being applied to big data include massively
parallel-processing (MPP) databases, search-based applications, data-mining grids, distributed
file systems, distributed databases, cloud based infrastructure (applications,
storage and computing resources) and the Internet.[citation needed]
Some but
not all MPP relational databases have the ability to store and manage petabytes
of data. Implicit is the ability to load, monitor, back up, and optimize the
use of the large data tables in the RDBMS.[52]
DARPA’s
Topological Data Analysis program seeks the fundamental structure of massive
data sets and in 2008 the technology went public with the launch of a company
called Ayasdi.
The
practitioners of big data analytics processes are generally hostile to slower
shared storage[citation needed],
preferring direct-attached storage (DAS) in its various forms from solid state
disk (SSD) to high capacity SATA disk buried inside parallel processing nodes.
The perception of shared storage architectures—SAN and NAS—is that they are
relatively slow, complex, and expensive. These qualities are not consistent
with big data analytics systems that thrive on system performance, commodity
infrastructure, and low cost.
Real or
near-real time information delivery is one of the defining characteristics of
big data analytics. Latency is therefore avoided whenever and wherever
possible. Data in memory is good—data on spinning disk at the other end of a FC
SAN connection is not. The cost of a SAN at the scale needed for analytics
applications is very much higher than other storage techniques.
There are
advantages as well as disadvantages to shared storage in big data analytics,
but big data analytics practitioners as of 2011 did not
favour it.[53]
Source: Wikipedia.
