Rift is an enterprise-grade reconciliation and discrepancy investigation engine written in compiled native Rust. It decouples high-throughput record alignment from underlying storage, enabling deterministic verification without full-table diffs.
Rift features two execution models: an ultra-fast in-memory reference engine for datasets fitting comfortably into available RAM, and an external sort/merge engine for massive datasets. The external reconciler sorts datasets into streaming chunk files on disk and executes a K-way merge, strictly respecting the user-configured --memory-limit-mb sorter allocation budget.
Differing database representations are mapped to Rift’s canonical type model: exact fixed-point decimals (preventing binary float inaccuracies), microsecond-precision ISO timestamps, bit-preserving IEEE-754 floats, UTF-8 strings, booleans, and explicit typed nullability. Normalization of Unicode, zero-values, and NaNs is policy-controlled.
Parameterized Graph Discrepancy Search (PGDS) transforms reconciliation from mere discrepancy reporting into automated triage. When records diverge, PGDS discovers candidate transformation rules (such as constant fee deductions, tax rate additions, or status remappings) and evaluates them via an authoritative D1 verification pass to isolate true anomalies.
Rift avoids proprietary black-box formats. Every run emits structured JSON artifacts detailing total processed records, exact match counts, delta attributes, and candidate verification reports. Exit codes (0 for clean match, 1 for discrepancies detected, 2 for operational errors) allow instant orchestration inside Airflow, cron, or GitHub Actions.
Explore how Rift CLI commands run against sample datasets.
In accordance with Eagle Eye’s rigorous scientific standards, we clearly separate what the RIFT engine demonstrates today from external integration boundaries and future engineering goals.
| System Area | Implemented & Verified Behavior | Operational Boundary & Limitations | Status |
|---|---|---|---|
| Input Formats & Readers | Native readers for JSONL, NDJSON, JSON array, CSV, TSV, Parquet, and PostgreSQL tables/views. | Warehouses (Snowflake, BigQuery, ClickHouse) are supported via approved exports or custom adapter development; not shipped native direct connectors. | Verified |
| Exact Record Matching | Primary & composite keys, explicit duplicate-key policies, unkeyed multiset comparison, column-level delta. | Schema and comparison policy must be explicitly specified; no implicit financial materiality or business tolerance is assumed. | Verified |
| Canonical Data Types | Exact decimals (fixed-point), typed ISO timestamps, bit-preserving IEEE-754 floats, strings, booleans, and typed nulls. | Unicode, NaN, and negative-zero normalization is policy-controlled, not silently universal. | Verified |
| Execution Engines | In-memory reference engine and external sort/merge engine with strict sorter budget (--memory-limit-mb). | Bounded to single machine / container host; no distributed multi-node cluster execution. | Verified |
| PGDS Investigation | Parameterized Graph Discrepancy Search candidate rule discovery, contrastive classification, and witness cache verification. | Discovers and independently verifies mathematical transformation rules; does not claim historical causation. | Verified |
| Runtime & Integration | Read-only access, immutable captures, local outbox delivery, CLI exit codes (0, 1, 2), and machine-readable JSON payloads. | Single-customer state boundary; no multi-tenant cloud SaaS or direct remote administration. | Verified |
We provide structured, read-only proof-of-concept evaluations for qualified enterprise engineering and settlement teams.