Need for Big Data

This is an article contributed by guest author, Sridhar Venkat, Data Architect

For sure, Big Data will be buzzword in the area of data architecture and data management for the next decade.

Need for the Big Data has risen because of the following:

The traditional OLTP databases are losing steam with the increasing demand in capturing and using the huge volume of data.

The delay in processing time to load the huge volume of real-time data and use them for analytics to derive business decisions based on the market conditions (for instance the Tics, prices in finance)

The traditional database lacks the functionality to handle the new variety of data like streams, videos, graphs etc

The development of big data was to address the above mentioned challenges and provides the features listed below

1. The scalable model for easy expansion of the Data nodes to accommodate growing needs to store the increasing data volumes
2. Spawning the process across the distributed cluster to reduce the load on the main name node
3. Mapping millions of records of data instead of the loading them through the ETL process
4. Ease of analysis of the real-time data, analytics and derive solutions using them

The “ACID” principles of the traditional databases do not apply to Big data.   Big Data maintains “eventual consistency”, since the change propagation is not instantaneous, there’s an interval of time during which some of the copies will have the most recent change, but others won’t.

In my view, the Big Data cannot be a standalone solution to any of the problems.  However, in association with the Traditional RDBMS it will be powerful tool to address the growing demands of the data needs in tomorrow’s technology.