Detect duplicate order references after trimming
Your task
Return normalized order_ref and copies for nonblank references that appear more than once. Sort by reference.
A clean key should identify one order. Group normalized keys to detect repeated imports.
Sample data
staged_orders| import_id | order_ref | status_text | amount_text |
|---|---|---|---|
| 1 | 201 | Completed | 2500 |
| 2 | 202 | COMPLETED | 1800 |
| 3 | 203 | completed | 3200 |
| 4 | 203 | completed | 3100 |
| 5 | 204 | REFUNDED | 5000 |
| 6 | 205 | completed | N/A |
| 7 | 206 | completed | 4000 |
| 8 | 207 | cancelled | 2400 |
| 9 | Completed | 1700 | |
| 10 | 209 | unknown | 2900 |
Preview of 10 of 12 rows
Expected output
| order_ref | copies |
|---|---|
| 203 | 2 |
Show a hint
Trim before grouping and use HAVING COUNT(*) > 1.
Data quality and reconciliation guide
Keep raw values available while you clean them
The staged_orders table is an import, so references, statuses, and amounts are text and may contain missing or malformed values. Trim whitespace before testing a field. An empty string after trimming can be turned into NULL to give missing data one representation.
SELECT import_id, NULLIF(TRIM(order_ref), '') AS clean_ref
FROM staged_orders
ORDER BY import_id;Compare records at the same grain
The payment ledger may have several charges or refunds for one order. Aggregate that ledger to one row per order before comparing it with shop_orders. Keep unmatched keys visible so a reconciliation reveals missing orders and missing payments.
SELECT order_id,
SUM(CASE WHEN kind = 'charge' THEN amount_cents ELSE -amount_cents END) AS net_cents
FROM payment_ledger
GROUP BY order_id
ORDER BY order_id;Worked solution
SELECT TRIM(order_ref) AS order_ref,COUNT(*) AS copies FROM staged_orders WHERE NULLIF(TRIM(order_ref),'') IS NOT NULL GROUP BY TRIM(order_ref) HAVING COUNT(*)>1 ORDER BY order_ref;Press Control+Space for SQL suggestions. Use the arrow keys to select an option and Enter to insert it.
Results
Loading SQLite…
