Why is Big Data a Buzzword at the Moment?

• Why is Big Data a buzz word at the moment?

For the first time, since the beginning of the informational revolution, we’re witnessing a surge of available data. This especially stands for the last decade when we’ve also experienced a significant increase in computational power, decrease in data storage costs, a huge progress in machine learning, and robust cloud solutions. This combination of factors allows any company to mine and benefit from its second most important resource after its people: data.

Google, Apache, IBM, etc. have been making use of their data than twenty years now. But the vast majority of the businesses are just now awakening for their digital transformation, trying to find ways how to dive into the data-driven world. 

That’s why everyone is talking about Big Data and how it will influence future data-driven decision making.

• How does Big Data work?

Big Data comes from many different data sources. That’s why, we need a mechanism to extract the data from all these sources, transform it into a standardized format easier to use and load it on a specific storage area, for example in the cloud, on-premise, or both. 

Afterwards, according to the value and veracity the data scientists need to prepare the data so it can be analyzed. Between 50% and 80% of the work is getting the data ready. At the end, it can be actually analyzed. No matter if it’s a simple algorithm or a complex neural network, most often the investment pays off.

• How do companies use Big Data and how do they benefit from it?

Big Data itself isn’t useful. These datasets are so voluminous that traditional data processing software cannot manage them. We need intelligent software. But these massive volumes of data can address business problems that we wouldn’t be able to tackle in a different way. By analyzing all the historical data, the companies can improve their product lines, cut manufacturing, logistics, marketing and sales costs, improve their efficiency, etc.

• What Big Data projects has Melon experience with?

Four years ago, we began an internal analytics project at Melon. It was based on our own data uploaded in the cloud. We used Microsoft Power BI to drill down and explore it. It helped us discover a lot of things that we were not fully aware of, and we were able to extract exact info out of the data that we have collected over the years. By using the findings, we were capable of standardizing our processes better, lower our costs, increase productivity and profitability. 

Our conclusion being that we can do better in data gathering, we developed and implemented new internal ERP software. We have used Microsoft Azure as a platform to store and manage all our internal data. We also integrated business intelligence tools to develop our analytics and reporting system. Now, Melon is using findings based on our historic data to make business decisions and take actions.

It was a great step for us to understand the future potential of data engineering and science, and that we needed to invest into our own technical team. Now, we differentiate a separate service offered to our clients. We’re currently working on multiple projects in the realm of the Big Data niche. The most interesting project that we are working on at the moment based on Big Data and Machine Learning is for a top Beveridge bottling company. We are helping our client with data engineering as part of their price prediction modeling. 

Melon is also delivering other projects using cutting edge technologies in the domains of IoT, Machine Learning and Blockchain. By leveling up our experience and internal know-how we are certain our Data Engineering and Science team will grow and will become sustainable and key part of our business.

• What are the world trends with Big Data? And what should we expect?

We are witnessing the evolution of different IaaS platforms in recent years. All these services evolve and mature in terms of storing and managing the data. They are designed to work more efficiently and effectively than traditional frameworks. Using these services where Data Engineers and Scientists can prepare, analyze and visualize big data is crucial for future Big Data evolution. By using and expanding these services, the companies will proceed their digital transformation process, so there is no doubt that this process will advance even faster than before. On the other side, IoT, AI, Machine Learning and Blockchain is feeding Big Data and pushing it forward to undiscovered territories. In recent years not only enterprise companies are using Big Data, instead more and more startups, small and medium companies heavily rely on this practice.

I also believe that IT companies will evolve even more by developing life-cycle management processes that can support the development of analytics apps and spread their products and services lines in this direction. So, the IT companies will keep on supporting this flow and we should expect an even greater expansion of Big Data and analytics as part of different domains as a lucrative way to grow.

• What are the challenges with the increasing volume of data?

Although new technologies have been developed for data storage, data volumes are doubling in size about every two years. This trend is about to expand even further, so organizations are still struggling to keep pace with their data and find ways to effectively store it. IoT connected devices are expected to reach a number of staggering 75 billion in 2025 from 27 billion devices currently, so it’s easy to see where that big data is coming from. The IaaS providers will continue to build new data centers so they can store and manage all this data. The best way around is to optimize the use of all that data on their servers. The organizations are losing huge money every year from the cost of storing data with poor quality. It remains to be seen how the organizations are going to address that.

• And what are the challenges when it comes to data security?

When talking about data security, we immediately get to General Data Protection Regulation that took effect in 2018. As we know, the GDPR is focused in Europe but at the same time some organizations, in an effort to simplify their business and promote good customer relations, have provided the same privacy protections for all their customers, regardless of their region. However, this is not the general position taken by all the businesses and organizations outside of Europe. The good news is that the biggest IaaS providers (Microsoft Azure, Amazon AWS, Google Cloud, IBM Cloud, etc.) are already designed with industry-leading security measures and privacy policies to safeguard the data in the cloud. This includes the categories of personal data identified by the GDPR. Furthermore, we have an existing trend of making hybrid clouds – a mix of IaaS (public) Clouds and on-premises (private) Clouds. Some organizations want to keep some sensitive data secure in its own data storage and the rest of the data to put in a rental Cloud storage. The companies are heavily investing in data security and this trend is here to stand.