The big data dilemma

Twenty years ago, software outsourcing was the key driver for economic prosperity for many developing countries and India had succeeded in dominates this market with over $100 billion dollars per year revenue. India's success formula was a lot of English speaking programmers and testers who were willing to work at a fraction of the cost of workers in developed countries. In this battle, China did not do well even it also had a lot of programmers and testers but they did not speak English as well as Indian's workers.

Five years ago, mobile applications were another key driver for economic prosperity for many developing countries. In this battle, there were several countries competed for the market estimated at $40 billion dollars per year revenue. So far India was also the key winner because it could change quickly by providing trainings to its programmers and testers in Apple's IOS and Google's Android platforms. However the battle to dominate the mobile apps market is still being fought as more tablets and new mobile platforms are being developed.

The next key driver for economic prosperity is likely to be Big Data where the market estimated to be worth over $150 billion per year revenue when global companies are looking for more skilled workers or countries that they can outsource to. Big Data is about information that companies have internally or collect from external sources such as internet and combining and analyzing them to get valuable information. Companies are using Big Data predict market trends to create business opportunities to grow and compete. This is a completely new area that requires more special trainings in data analysis and business intelligence. Since this is so new, few universities would even have training programs or professors to teach it, so it is a wide open opportunity for anyone. The industry predicts that they need at least 200 thousand Data scientist by 2015 so there is a rush to develop more data scientists among countries such as India and China as well as other developing countries. Both India and China government have declared that this is a key priority in training to have at least 50,000 Data scientists in the next five years.

Data Scientists are software developers with advance training in mathematics, statistics, artificial intelligence, and machine learning. They work on all types of data to predict the trend that can help company to plan for the future. Since this is a multi-disciplinary skill, and requires at least a Master degree in Big Data analysis, it is very rare to find enough data scientists in any country. Currently India is emerging as the most aggressive country to develop workers in Big Data solutions but according to industry report, there are many other countries are also developing trainings to build up a strong workforce to compete for this lucrative market. The report states that globally, companies generated and consumed 1.8 zettabytes of data in 2011 and is expected to grow to 35 zettabytes by 2015. (A zettabyte is a trillion gigabytes, or a billion terabytes.) And they will need more data scientist than India and China can provide in the next five years. Currently, there is a shortage of 100 thousand trained Data Scientists globally and the shortage continues to increase in the next few years.

The business of Big Data is already reaching a critical point in Western countries where Big Data analysis is shaping up strategies for many global companies. A Wall street analyst said: “Whoever controls Big Data will control the entire software market as it is bigger than all software development and mobile applications combine and it could grow to the biggest market worth trillion dollars in the next ten years.” That is why Microsoft, Google, Facebook, IBM, Oracle, and Amazon are placing their future on this new area but the main question is: Where can they find workers?


  • Blogs of Prof. John Vu, Carnegie Mellon University
Don't forget to follow us on Facebook!