TL:DR — KISS 💕 DevOps!
Sparked by a discussion at work, I am trying to write down my personal perspective on what best-practices for data engineering look like. To do that I thought it would be best to first define the goal of data engineering by answering the question:
What is the outcome that we, the Data Engineers, are trying to drive?
And then, after we are clear on the outcome, let us detail which principles and practices can help us drive that.
A disclaimer: I have working experience in what I would call big enterprises (> 50.000 employees), working…
If you work in analytics, it is safe to assume that you have complained about data quality issues more than once. We all have, and the pain is real!
As a Data Engineer, data and its quality are close to my heart. I am acutely aware of how paramount good data quality is — and recently this topic has created some buzz with e.g. #DataOps. …