Automated data workflows often complete with status SUCCESS while silently dropping records, truncating strings, or corrupting key metrics. Rift introduces independent, deterministic data verification for modern engineering stacks.
Standard data observability monitors job runtimes, row counts, and CPU spikes. But row-count equality does not verify that row contents match the agreed reference.
Migrating from PostgreSQL or Oracle to modern cloud data lakes? Rift compares live source tables against target Parquet files record-by-record, verifying 100% data fidelity before switching live queries.
Refactoring dbt SQL transformations or updating PySpark jobs? Run Rift as a validation step in CI/CD to compare staging outputs against production snapshots. Catch logic regressions before merge.
Raw event streams frequently lack unique primary keys. Rift’s --unkeyed multiset reconciler uses optimized hashing to verify multiset equality in $O(N)$ time without $O(N^2)$ quadratic freezes.
Rift requires no daemon, no persistent control plane, and no database credentials stored in external SaaS clouds. Invoke it directly as a step inside Apache Airflow, Prefect, GitHub Actions, or Kubernetes Jobs.
# Execute read-only verification before publishing table rift compare \ s3://lake/raw/staging_20261022.parquet \ postgres://prod_db/curated_events.json \ --schema /opt/rift/schema.json \ --key event_id,timestamp \ --engine external \ --memory-limit-mb 64 \ --output json > /tmp/reconcile_report.json # Exit code 0 ensures clean propagation if [ $? -eq 0 ]; then echo "Data verification passed. Publishing to warehouse." fi
Benchmark Rift on your largest Parquet and database extracts under an engineering evaluation agreement.