Velocity of Big Data
A developer wrote to me: “I enjoy your blogs on Big Data. I understand the volume of Big Data; the variety of data such as structured and unstructured, but I do not understand the velocity issue. Please explain.”
Answer: There are three key elements of Big Data (The three V: Volume, Variety and Velocity.) Velocity means the data changes and grows fast. It is getting bigger and bigger with time. For example, people add data into their websites all the time and it grows quickly. Stock market transactions move billion or trillions of dollars every few seconds. Weather sensors measure change in temperature every few minutes. YouTube adds streaming videos every five minutes. Not only these data requires huge storage but it changes so fast and makes the analysis so difficult. You must collect and analyze these data in “Real time” or else it loses the meaning. For example, if you monitor stock market trends based on yesterday data then you may miss something due to your “out of date” information.
The challenge of Big Data, not just it is huge, with all types of data, but it is also changing very fast. If you cannot collect, control, analyze, and make decision on it then somebody will. In global competition business, that is the difference between winner and loser. Big data analysts must ask these questions: How often do we analyze these data? Every ten minutes? Every hour? Or every day? When is the data being invalid? What are we going to do with out of date data? How do we know?
Sources
- Blogs of Prof. John Vu, Carnegie Mellon University
