Templates

Offer Snapshots

Data Contract Template

Ensure data from merchant offer snapshots is fresh, complete, and reliable before it is used for pricing operations, market availability analysis, and competitive benchmarking.

Data contract description

This data contract is designed for Idealo, where daily offer snapshots capture merchant pricing, availability, and market data across supported countries. By governing the offer_snapshots dataset with a composite primary-key uniqueness check on offer_id and date_utc, a 24-hour freshness window, and market- and enum-level constraints, the contract helps Idealo keep offer state data dependable for downstream pricing operations, availability monitoring, and competitive benchmarking.

Dataset

Defines the dataset this contract applies to, including its source and scope. This is the single point of reference for what data is being validated and protected.

Dataset checks

Validations that apply to the dataset as a whole, including schema stability, freshness, record presence, and cross-row integrity. This contract includes a schema check that disallows extra columns and unexpected column reordering, a row_count check to prevent empty loads, a dataset-level uniqueness check on the composite key formed by offer_id and date_utc, and a freshness check requiring date_utc to be less than 24 hours old.

Column checks

Column-level rules that define required fields, uniqueness, allowed formats, and value constraints for individual attributes. This contract enforces that offer_id, date_utc, price, currency, availability_status, merchant_id, and country are not missing, and it requires offer_id to be unique. It constrains price to be non-negative, and restricts currency, availability_status, and country to approved values.

offer_snapshots_data_contract.yaml

dataset

variables:
  ASSET_NAME:
    default: "offer_snapshots"
  ASSET_TYPE:
    default: "merchant_offer_feed"
  BUSINESS_DOMAIN:
    default: "Offers"
  OWNER_TEAM:
    default: "Pricing Team"
  ONCALL_ROUTE:
    default: "#pricing-support"
  DATA_DICTIONARY_REFS:
    default: |
      offer_id -> dict.offer_id.v3
      price -> dict.price.v2
  STORAGE_LOCATION_BASE_PATH:
    default: "s3://idealo-offers/state/"
  PARTITIONING_SPEC:
    default: "date_utc, country"
  SLA_DUE_TIMES:
    default: "00:15 UTC daily"
  PRIMARY_KEYS_AND_UNIQUENESS:
    default: "offer_id, date_utc"
  SCHEMA_SPEC:
    default: "offer_id: string, date_utc: string, price: float, currency: string, availability_status: string"
  CONSTRAINTS_AND_ENUMS:
    default: "price >= 0, currency IN (USD, EUR, GBP), availability_status IN (in_stock, out_of_stock, preorder, unknown)"
  FRESHNESS_AND_COMPLETENESS_EXPECTATIONS:
    default: "≥90% of active merchants present by SLA due"
    
checks:
  - schema:
      allow_extra_columns: false
      allow_other_column_order: false
  - row_count:
      threshold:
        must_be_greater_than: 0
  - failed_rows:
      name: "Ensure primary keys are unique"
      qualifier: pk_uniqueness
      query: |
        SELECT offer_id, date_utc
        FROM datasource.database.schema.offer_snapshots
        GROUP BY offer_id, date_utc
        HAVING COUNT(*) > 1
      threshold:
        must_be: 0
  - freshness:
      column: date_utc
      threshold:
        unit: hour
        must_be_less_than: 24

columns:
  - name: offer_id
    data_type: string
    checks:
      - missing:
      - duplicate:
  - name: date_utc
    data_type: string
    checks:
      - missing:
  - name: price
    data_type: float
    checks:
      - missing:
      - invalid:
          name: "Price must be non-negative"
          valid_min: 0
  - name: currency
    data_type: string
    checks:
      - missing:
      - invalid:
          name: "Valid Currencies"
          valid_values:
            - USD
            - EUR
            - GBP
  - name: availability_status
    data_type: string
    checks:
      - missing:
      - invalid:
          name: "Valid Availability Status"
          valid_values:
            - in_stock
            - out_of_stock
            - preorder
            - unknown
  - name: merchant_id
    data_type: string
    checks:
      - missing:
  - name: country
    data_type: string
    checks:
      - missing:
      - invalid:
          name: "Supported Markets"
          valid_values

Data contract description

Dataset

Defines the dataset this contract applies to, including its source and scope. This is the single point of reference for what data is being validated and protected.

Dataset checks

Column checks

offer_snapshots_data_contract.yaml

dataset

variables:
  ASSET_NAME:
    default: "offer_snapshots"
  ASSET_TYPE:
    default: "merchant_offer_feed"
  BUSINESS_DOMAIN:
    default: "Offers"
  OWNER_TEAM:
    default: "Pricing Team"
  ONCALL_ROUTE:
    default: "#pricing-support"
  DATA_DICTIONARY_REFS:
    default: |
      offer_id -> dict.offer_id.v3
      price -> dict.price.v2
  STORAGE_LOCATION_BASE_PATH:
    default: "s3://idealo-offers/state/"
  PARTITIONING_SPEC:
    default: "date_utc, country"
  SLA_DUE_TIMES:
    default: "00:15 UTC daily"
  PRIMARY_KEYS_AND_UNIQUENESS:
    default: "offer_id, date_utc"
  SCHEMA_SPEC:
    default: "offer_id: string, date_utc: string, price: float, currency: string, availability_status: string"
  CONSTRAINTS_AND_ENUMS:
    default: "price >= 0, currency IN (USD, EUR, GBP), availability_status IN (in_stock, out_of_stock, preorder, unknown)"
  FRESHNESS_AND_COMPLETENESS_EXPECTATIONS:
    default: "≥90% of active merchants present by SLA due"
    
checks:
  - schema:
      allow_extra_columns: false
      allow_other_column_order: false
  - row_count:
      threshold:
        must_be_greater_than: 0
  - failed_rows:
      name: "Ensure primary keys are unique"
      qualifier: pk_uniqueness
      query: |
        SELECT offer_id, date_utc
        FROM datasource.database.schema.offer_snapshots
        GROUP BY offer_id, date_utc
        HAVING COUNT(*) > 1
      threshold:
        must_be: 0
  - freshness:
      column: date_utc
      threshold:
        unit: hour
        must_be_less_than: 24

columns:
  - name: offer_id
    data_type: string
    checks:
      - missing:
      - duplicate:
  - name: date_utc
    data_type: string
    checks:
      - missing:
  - name: price
    data_type: float
    checks:
      - missing:
      - invalid:
          name: "Price must be non-negative"
          valid_min: 0
  - name: currency
    data_type: string
    checks:
      - missing:
      - invalid:
          name: "Valid Currencies"
          valid_values:
            - USD
            - EUR
            - GBP
  - name: availability_status
    data_type: string
    checks:
      - missing:
      - invalid:
          name: "Valid Availability Status"
          valid_values:
            - in_stock
            - out_of_stock
            - preorder
            - unknown
  - name: merchant_id
    data_type: string
    checks:
      - missing:
  - name: country
    data_type: string
    checks:
      - missing:
      - invalid:
          name: "Supported Markets"
          valid_values

Data contract description

offer_snapshots_data_contract.yaml

dataset

variables:
  ASSET_NAME:
    default: "offer_snapshots"
  ASSET_TYPE:
    default: "merchant_offer_feed"
  BUSINESS_DOMAIN:
    default: "Offers"
  OWNER_TEAM:
    default: "Pricing Team"
  ONCALL_ROUTE:
    default: "#pricing-support"
  DATA_DICTIONARY_REFS:
    default: |
      offer_id -> dict.offer_id.v3
      price -> dict.price.v2
  STORAGE_LOCATION_BASE_PATH:
    default: "s3://idealo-offers/state/"
  PARTITIONING_SPEC:
    default: "date_utc, country"
  SLA_DUE_TIMES:
    default: "00:15 UTC daily"
  PRIMARY_KEYS_AND_UNIQUENESS:
    default: "offer_id, date_utc"
  SCHEMA_SPEC:
    default: "offer_id: string, date_utc: string, price: float, currency: string, availability_status: string"
  CONSTRAINTS_AND_ENUMS:
    default: "price >= 0, currency IN (USD, EUR, GBP), availability_status IN (in_stock, out_of_stock, preorder, unknown)"
  FRESHNESS_AND_COMPLETENESS_EXPECTATIONS:
    default: "≥90% of active merchants present by SLA due"
    
checks:
  - schema:
      allow_extra_columns: false
      allow_other_column_order: false
  - row_count:
      threshold:
        must_be_greater_than: 0
  - failed_rows:
      name: "Ensure primary keys are unique"
      qualifier: pk_uniqueness
      query: |
        SELECT offer_id, date_utc
        FROM datasource.database.schema.offer_snapshots
        GROUP BY offer_id, date_utc
        HAVING COUNT(*) > 1
      threshold:
        must_be: 0
  - freshness:
      column: date_utc
      threshold:
        unit: hour
        must_be_less_than: 24

columns:
  - name: offer_id
    data_type: string
    checks:
      - missing:
      - duplicate:
  - name: date_utc
    data_type: string
    checks:
      - missing:
  - name: price
    data_type: float
    checks:
      - missing:
      - invalid:
          name: "Price must be non-negative"
          valid_min: 0
  - name: currency
    data_type: string
    checks:
      - missing:
      - invalid:
          name: "Valid Currencies"
          valid_values:
            - USD
            - EUR
            - GBP
  - name: availability_status
    data_type: string
    checks:
      - missing:
      - invalid:
          name: "Valid Availability Status"
          valid_values:
            - in_stock
            - out_of_stock
            - preorder
            - unknown
  - name: merchant_id
    data_type: string
    checks:
      - missing:
  - name: country
    data_type: string
    checks:
      - missing:
      - invalid:
          name: "Supported Markets"
          valid_values

How to Enforce Data Contracts with Soda

Embed data quality through data contracts at any point in your pipeline.

# pip install soda-{data source} for other data sources

pip install soda-postgres

# verify the contract locally against a data source

soda contract verify -c contract.yml -ds ds_config.yml

# publish and schedule the contract with Soda Cloud

soda contract publish -c contract.yml -sc sc_config.yml

Check out the CLI documentation to learn more.

How to Automatically Create Data Contracts.
In one Click.

Automatically write and publish data contracts using Soda's AI-powered data contract copilot.

Make data contracts work in production

Business knows what good data looks like. Engineering knows how to deliver it at scale. Soda unites both, turning governance expectations into executable contracts.

Book a demo