top of page
Writer's pictureAriel Menche

Data Piping


What is Data Piping
People talking in an office possibly about data piping

One time we had a massive garbage collection client in Chicago.

They flew us and 87 consultants to Chicago to tackle one problem: product profitability.


The company had developed and acquired dozens of products over the years. Now their revenue was plateauing. They needed to become strategic about growth.


But first they had a major hurdle to overcome: they had zero insight into which of their products were profitable.


Fun fact: this was not a mom-and-pop shop. It was a Fortune 500. If they were struggling to determine profitability, you better bet tons of smaller companies do too.



Why were they struggling with product profitability?


Their source data was a mess. An absolute mess.


Product lines were completely siloed. Each had its own records and data collection methods. Some used sophisticated software while others relied on handwritten notes from drivers (!!!)

The result of these methods? A patchwork of data that was effectively impossible to consolidate and analyze.


Our job was not just to sort and analyze the existing data, but establish systems and processes to ensure the continued collection of accurate, relevant and streamlined information.

This might sound obvious now, but the ability to process Big Data is fairly new. When I worked with this client a couple years ago, no one expected companies to operate unified systems of data management. Information was scattered and that was normal. Different departments handled data in different formats, with duplicates and outdated records being common.


Obviously that’s a massive issue for many reasons. One of the big ones is that it prevents a company from viewing a 360 degree view of their company and make strategic decisions with this information. The client was completely blind to product profitability. How could they even start deciding how to grow?



Data Piping

How can you avoid this problem? If you’re a CEO or CFO of a growing company, you’ve probably thought hard about data collection and management. Not only for regulatory reasons, but your own benefit.


First, two terms: Data Pipeline and Data Integration.


Effectively these are the same thing: the process of collecting raw data from multiple sources and organizing it into a clean database.

There are 4 most common types of data sources that collect data:

  • HRIS or Human Resources Information System: systems that maintain data on employees, payroll, and other matters relating to the people of an organization.

  • ERP or Enterprise Resource Planning - systems that help run your entire business, including Finance, HR, Supply Chains and HR, Budgeting, Inventory and more.

  • CRM or Customer Relationship Management - platforms that helps you manage of your company’s relationships and interactions with customers and prospects.

  • Internal spreadsheets and databases used to collect information across the business


Once raw data is imported, it needs to be processed. That’s because each data source can be distorted by overlapping information, inconsistent terminology and other contradictions.


Processing includes 3 stages:

  • Cleaning - Removing duplicates, handling missing values and correcting errors

  • Transformation - Converting data types, aggregating information, etc.

  • Standardization - Ensuring consistency in terminology, units, etc.


It’s best to preempt these issues by implementing quality control processes so they never happen. That way data can maintain its integrity and you can avoid deviations from expected patterns in advance.


The ideal way to do this is by ensuring that data collection, processing and display happens in real-time. Instead of batch processing—where data is collected and processed at predefined intervals—real-time processing empowers your organization to respond to events and changes instantly. Of course, real-time processing is company-dependent and may not be right for your organization.


Another layer of data piping is security and data accessibility. Data should be easily accessible to authorized personnel while maintaining security and privacy. One key security measure is RBAC or Role-Based Access Control, which allows you to assign specific access to users based on their responsibilities and functions within the organization.



analyzing data

Benefits

Once you streamline your source data and process into one consolidated database, you become empowered by becoming able to leverage these data points and therefore optimize important decisions.


Information is data with context. Data are points, while information is a story

There are tons of use cases here. Unlimited, really. Let’s return to the waste management example above. As you can tell, our client experience revealed huge opportunities around pricing. There were major inconsistencies, and only once the analysis was complete were they were able to unify pricing strategy and increase both top-line revenue and gross margins. Since the conclusion of this project, their quarterly revenues have nearly doubled.


Maybe you’re thinking about expanding a product line. Wouldn’t you like to know if that product is profitable first? That requires data: clean, streamlined data. And you know what’s even better? Getting that data in real time, so you can make decisions on the go about whether to expand or constrict certain products by the week or quarter, and not by the year.


With streamlined data you also avoid massive mistakes, like overbudgeting or underbudgeting, and so much more.


In the context of strategic finance, effective Data Piping is the backbone of strong decision-making. It provides a streamlined flow of information: from gathering, processing and organizing data to ensuring the right people have access to it at the right times.


Summary

  • A model is only valuable as much as its source data. Set up preemptive ways to collect and process data so you can react in real time and leverage the necessary information to make active strategic decisions.

  • Data Piping is essential for strategic finance. Strategic actions like optimizing investments, funding, and operational strategies to enhance the bottom line require the integrity of the numbers.

  • This is also super important if you plan to exit. Prepare for selling your company far in advance by ensuring that your data is clean and organized. And do this before it’s too late!

Comments


InSights

bottom of page