Data: Your white truffle

At PYGIO, we spend a fair amount of time thinking about data: how it’s generated, handled and used by our clients. We thought we’d write a short-form post explaining all things data-related, from the tools, frameworks and disciplines involved in proper data management.

Setting the Scene:

Our modern world is not just inundated with data; it’s virtually submerged in it. It is estimated that 90% of the world’s data was generated in the last two years alone. According to the best estimates, 328.77 million terabytes of data are created each day. If the average size of a book is assumed to be roughly 1 megabyte, then 328.77 million terabytes is equivalent to 328.77 quadrillion books. That’s about 4,360 times the estimated number of books ever written in human history – generated in a single day!

Data Science

Beneath this overwhelming avalanche of data lies a treasure trove of hidden insights. Companies sit on vast amounts of raw data – transactions, customer interactions, market trends, and more. Yet, without the right tools and methodologies, this data remains a dormant white truffle: valuable but underutilised. 

Data science is a systematic approach that companies use to make sense of and extract meaningful insights from this complex data. It’s the foundation for informed decision-making through data wrangling and modelling. 

A practical example of this would be a retail company that starts to analyze its past sales data, customer reviews, and inventory levels. Using data science techniques, it identifies which products sell best in different seasons, how promotions affect sales, and what customers say about their products. The company uses these insights to stock its stores better, plan sales, and improve product offerings.

Predicting the Future: Machine Learning 

As companies mature on their data journeys, they aim to predict future trends and behaviors. Machine learning is a subset of data science where algorithms are trained on historical data to make such predictions. It automates analytical model building, enabling businesses to respond dynamically and plan for changing scenarios. Machine learning allows humans to see data in new ways, often giving the ability to see in multiple hundreds of dimensions, when humans are only limited to three.

The retail company now wants to anticipate future demand based on the above example. It employs machine learning to predict which products will likely be best-sellers next season. Based on past purchase patterns, the ML model also recommends products to online shoppers, increasing the likelihood of additional sales. Furthermore, the model can alert when inventory levels are likely to run low, helping the company to restock efficiently.

Artificial Intelligence

As more processes within the company become digitised, it is possible to further optimise business processes with end-to-end decision making. Intricate tasks that once took teams of specialists, get automated, complex decisions are made, and human-level interactions become replicated.

Going back to the practical example, the company now decides to implement AI-driven chatbots to assist online customers in real-time, guiding their shopping experience based on their past behaviors and current trends. In their physical stores, smart cameras analyze customer foot traffic to optimize store layouts. The AI systems continuously learn, refining their operations to enhance the overall customer experience and operational efficiency.

Laying the Foundation with proper Data Governance

Before a company embarks on building out its Data Science & ML capability, a proper foundation needs to be set. Proper Data Governance is this foundation. At its core, data governance is about managing and ensuring the quality, security, and proper usage of a company’s data. 

By ensuring high-quality, reliable data, Data Governance Frameworks lay the groundwork for effective Data Science and ML applications. This foundation means that their insights are based on trustworthy data, leading to more accurate predictions and better decision-making.

Beyond the technical intricacies, data governance also plays a role in building a culture of data accountability and transparency within an organisation. It’s not just about having the right tools and protocols in place; it’s about instilling a mindset where every stakeholder understands their role in the data ecosystem. In simple terms, when everyone in a company, from the CEO to the newest staff member, follows the rules and guidelines set for handling data. It makes sure that the information remains accurate and trustworthy from the moment it’s collected until it’s no longer needed.

Wrapping it all together: The Data Strategy

All of the above concepts are rationalised with business objectives through a higher-order framework: the Data Strategy. This refers to the broader vision or plan that an organisation adopts concerning its data. It’s about how data is prioritised as a first-class citizen to achieve business goals. 

While Data Governance ensures that data is trustworthy and managed correctly, the Data Strategy determines how data will drive business value, innovation, and market differentiation. It is important that companies invest in understanding and implementing these frameworks to see the best returns. 

AI Ethics, Bias and Data Privacy

Before embarking on a Data Strategy, companies need to understand the ethical challenges at play when operationalizing their data. At the forefront of these are issues of Bias and Data privacy. Bias can perpetuate existing prejudices, making fairness in AI an important topic. Data privacy surrounds the safeguarding of personal information, ensuring that individuals’ rights are upheld and their data isn’t misused, exposed, or accessed without proper authorization.

For companies leveraging these automated tools, it’s crucial that they ensure their data governance policies are compliant with privacy and fairness best practices and laws (for example POPIA in South Africa). This means not only protecting individual’s data from breaches but also being transparent about how their data is used in the decision-making process.

In Closing

From understanding the value of your company’s data to deploying sophisticated AI-driven tools, the overarching theme remains consistent: companies that tap into their data assets are able to sustainably drive innovation and growth.

As the great Tim O’Reilly said, “ Data is not the new oil. It is the new sand. Like silicon, which makes up 28% of the earth’s crust, it is ubiquitous & is only made valuable by an enormous set of industrial-scale processes.”

Choosing these processes wisely sets your company up for long-term, sustainable success.

Eager to unpack how a proper Data Strategy can transform your business? Feel free to contact Head of Data Science Nicolas Gonatas

More articles from the PYGIO team:

• How to improve your software development process – a project manager’s perspective
• Software 2.0
• The power of storytelling in sales
• Challenges and opportunities in banking and fintech by Ihor Medvedkov
• To be simple or not to be simple – that is the thing
• How to handle a breakup with a client
• Serverless vs Frameworks Architecture