Start on the Right Before Shifting-left

Maarten Masschelein

CEO and Founder at Soda

The other day I overheard an analyst complaining that the upstream data engineering team was producing bad data. In turn, these data engineers blamed upstream software engineers for shipping breaking changes. Sounds familiar?

By now, most of us know that improving data quality is a team sport. Instead of pointing fingers, data teams must instead spend their energy on improving cross-team communication, with the aim to prevent mistakes from happening in the first place.

So how and where to start? Start right and shift-left where it makes sense. Capture business requirements through an easy-to-use interface, and shift those requirements left to be able to circuit-break new data (Airflow) or code release (Github).

Start Right, Because Your Customer is King/Queen

Not all data is created equal; only a few failures ultimately require your immediate attention. To determine what matters, you must involve your customers (that’s data product thinking) to set data quality requirements and SLOs. In order for data producers to test and monitor what matters, they need to know what their customers are looking for.

To protect you against data issues, Soda introduces its no-code interface for business collaboration. Generally Available today in Soda Cloud. Soda’s no-code environment makes it easy for data product teams to put their customers in the driving seat for defining data quality checks. Have a look for yourself.

Shift-left to the Source

Running data quality checks outside of your pipeline is a great starting point to monitor and observe changes, but monitoring only tells you when things are too late. Where data observability ends, shift-left data governance begins.

Use Soda’s Data Contracts in combination with the no-code user interface to solve Data Governance end-to-end. Integrate Soda directly into your Orchestration tool (Airflow) to circuit-break bad data from transposing into your data warehouse. Integrate Soda into your CI process (Github) to prevent bad code releases from creating breaking changes.

Data product owners/managers should drive the definitions of data contracts based on the use-cases and business requirements. Use Soda Cloud to define these requirements, and shift those left to the data producers where it matters. Soda helps you find the data worth fixing.

Get Started With No-code Checks and Data Contracts Today!

New to Soda? Register for a free Soda Cloud account.

Existing customers and users, sign in here.

The self-serve account creation for Soda Cloud is temporarily paused as we're preparing the general availability of several major updates. If you want to try Soda Cloud in the meantime, please schedule a call with our team of experts, discuss your use case, and get started.