How Array Matching Works
JSON arrays don't tell you which objects represent the same record. Most diff tools compare index 0 to index 0, index 1 to index 1, and so on. That works until items are reordered, inserted, or removed. Before comparing array objects, JSON Semantic Diff first solves a correspondence problem: which objects represent the same entity?
1. Why Positional Matching Breaks
In standard text-based and naive structural diff tools, arrays are compared index-by-index: element [0] is compared against [0], [1] against [1], and so on.
In real-world data systems—such as microservice API responses, database dumps, and sorted configuration files—record arrays are frequently reordered, paginated, or prepended. Positional diffing breaks under these ordinary operations:
If an array with 1,000 records has a single new item prepended at index 0, positional matching compares record 0 with record 1, record 1 with record 2, and so on. Instead of reporting 1 insertion, the tool reports 1,000 modified records—obscuring every genuine change beneath false noise.
2. Identity Candidate Evaluation
Databases enforce primary keys so records have identity regardless of physical row order. Raw JSON documents, however, rarely specify which field is the primary key.
JSON Semantic Diff discovers scalar candidate fields across the array elements (such as id, sku, user.email) and evaluates their statistical behavior across both documents without requiring hardcoded schemas or external configuration.
3. Composite Identities (Inventory by Store)
In many datasets, no single field is unique on its own. For example, in a multi-location inventory table:
storealone has duplicates across records (multiple items exist inBOS).skualone has duplicates across records (the sameSKU-1001exists in multiple stores).store + skutogether is unique across all records and forms an unambiguous composite identity.
When no single field achieves identity quality, the engine evaluates multi-field combinations. A complexity penalty of -3% per extra field ensures that simpler single-field keys are always preferred when both perform equally well.
4. Additions, Removals & Match Coverage
Match Coverage (27% weight) measures the fraction of records in the smaller side that have a matching counterpart on the other side: min(leftCount, rightCount).
By measuring against the smaller side rather than total records, ordinary array growth (such as appending a new customer) or record deletions do not erode the key's score.
5. Why Population Overlap (Jaccard) Is Also Needed
Relying solely on Match Coverage creates a dangerous blind spot: comparing 3 records against 300 unrelated records where 3 happen to share key values would yield 100% coverage.
Population Overlap (15% weight) computes the Jaccard set similarity across all distinct keys: |A ∩ B| / |A ∪ B|. Population overlap reduces the score in extreme population-mismatch cases and helps prevent unsafe automatic matching.
6. Candidate Score & The Six Scoring Signals
Candidate score measures how well a field or field combination behaves as an identity. Every candidate key is evaluated against six scoring signals that sum to a 1.0 (100%) base weight:
Are values distinct within each side? Measured as the average of left and right uniqueness ratios.
What fraction of records in the smaller array find a counterpart on the other side?
The Jaccard similarity of distinct key values across both sides (|A ∩ B| / |A ∪ B|).
Is the candidate key present, non-null, and non-empty across every record in both arrays?
Are key values consistently the same primitive JavaScript type (e.g. all strings or all numbers)?
A secondary signal for identity-like names (id, uuid, sku). At 6%, it sits above the 5% ambiguity margin, allowing it to resolve ties between otherwise equal candidates.
- Complexity Penalty (-3% per extra field): Applied to composite keys (e.g.
store + skuhas 1 extra field = -3%). Prefers simpler keys when both perform equally well. - Volatility Penalty (-3%): Applied if a field name suggests mutable state (
timestamp,updatedAt,price,status).
7. Confidence, Runner-Up, Margin & Ambiguity
Candidate score measures how well a field or field combination behaves as an identity. Confidence additionally considers safety gates and how clearly the best candidate beats the runner-up.
- Best candidate: The highest-scoring identity candidate (
bestScore). - Runner-up: The second-highest scoring candidate (
runnerUpScore). - Margin:
bestScore − runnerUpScore.
A high score is not enough if another candidate is almost equally convincing. In ambiguous cases, JSON Semantic Diff does not guess.
8. Safe Fallback & Decision Gates
Clearing a raw blended score is not enough. Before an inferred key is automatically applied to reorder and pair array records, it must satisfy strict safety gates:
| Decision Gate | Criteria | Engine Action |
|---|---|---|
| Auto-Apply (High) | Score ≥ 90.0%, Margin ≥ 5.0% over 2nd best candidate, Match Coverage ≥ 70.0%, and Uniqueness ≥ 95.0% on both sides. | Automatically pairs array items by identity. |
| Medium Confidence | Score ≥ 75.0%, Margin ≥ 3.0%, Match Coverage ≥ 50.0%, and Uniqueness ≥ 95.0% on both sides. | Offered to the user with visible confidence indicator. |
| Ambiguous | Two candidates both score ≥ 75.0% and margin < 5.0%. | Refuses to guess; alerts user of multiple competing candidates. |
| Below Threshold | No candidate clears the confidence gates. | Safely falls back to positional index-by-index matching. |
9. Deterministic by Design
The same inputs and options produce the same matching decision. JSON Semantic Diff does not use embeddings, fuzzy matching, or LLM calls to infer identity.
Identity is inferred primarily from how fields behave in the data, not from a hardcoded list of property names. Field-name hints may contribute as a secondary signal, but the matching decision is primarily data-driven.
10. Live Engine Evaluation: Inventory by Store
The example below runs the live diffJson comparison engine on the built-in Inventory by Store dataset. Neither store nor sku is unique by itself, but together they form an unambiguous identity key.
Ready to explore semantic diffing?
Try the Inventory by Store example in the diff workbench or paste your own JSON documents.