Data Engineering | Data Catalyst Q&A with Ranjan Bhattacharya & Ed Lyons
Click on the Image above to watch the YouTube Playlist of EQengineered’s Data Engineering | Data Catalyst Q&A with Ranjan Bhattacharya & Ed Lyons
· What are the recent trends in data management?
Building a data pipeline was an enormous on-prem custom job. It needed multi-year investments, and sizable teams to pull one off. The more traditional on-prem data warehouse tools needed data to be beaten into submission before it could be brought in. There were severe constraints in the volume and the variety of data that was acceptable to these tools.
Modern cloud-based data stacks have completely upended the cost of building a data pipeline. It is now possible to start small, and build a pipeline very inexpensively, focused on a narrow section of the business. The tools are more configurable, flexible, and forgiving of data formats.
· How should organizations go about modernizing their data and analytics architecture?
The days of organizations embarking on projects spanning multiple years, and tens of millions of dollars, are long gone. Instead, in the Agile economy, it makes more sense to have short initiatives, focused on solving a few critical problems, and building data and analytics capabilities more iteratively.
A short, focused data readiness catalyst engagement, typically 6—12 weeks long, like the one offered by EQengineered, can help organizations identify data readiness gaps and provide a roadmap to build their data and analytics maturity.
What can go wrong if the enterprise doesn’t have access to the right data?
Good, clean data is essential for business leaders to glean meaningful and accurate insights; for example, how the business is running, where it is functioning well, and where it is falling behind.
It is common for different applications—sales, marketing, service, account management, etc.—to have different representations for a single customer. Unless these different views are consolidated, the business will not have a good way to identify who they are selling to, which are the most profitable customers, and how much the business is spending to service its customer(s). Without proper data practices, parts of the business may become invisible.
Why should an enterprise invest in data modernization prior to modernizing applications?
It can be best explained by using a cooking metaphor. Good cooking is a function of not only the appliances, like your oven or air fryer, but more importantly of healthy, fresh ingredients.
For an organization, most people interact with various applications for individual business functions like sales, inventory management, payroll, expense tracking, etc. These are like the appliances in a kitchen.
However, these applications need good data to function and to generate usable data in turn, which needs to be consolidated in a central place from which to draw insights.