Keep the latest imported row per order
Your task
For each nonblank trimmed order_ref, return the normalized reference and the import_id of its latest row. Use the highest import_id as latest. Sort by reference.
Deduplication needs an explicit rule for deciding which copy to keep.
Sample data
staged_orders| import_id | order_ref | status_text | amount_text |
|---|---|---|---|
| 1 | 201 | Completed | 2500 |
| 2 | 202 | COMPLETED | 1800 |
| 3 | 203 | completed | 3200 |
| 4 | 203 | completed | 3100 |
| 5 | 204 | REFUNDED | 5000 |
| 6 | 205 | completed | N/A |
| 7 | 206 | completed | 4000 |
| 8 | 207 | cancelled | 2400 |
| 9 | Completed | 1700 | |
| 10 | 209 | unknown | 2900 |
Preview of 10 of 12 rows
Expected output
| order_ref | import_id |
|---|---|
| 201 | 1 |
| 202 | 2 |
| 203 | 4 |
| 204 | 5 |
| 205 | 6 |
| 206 | 7 |
| 207 | 8 |
| 209 | 10 |
| 210 | 11 |
| 999 | 12 |
Show a hint
Group by the trimmed reference and select MAX(import_id).
Data quality and reconciliation guide
Keep raw values available while you clean them
The staged_orders table is an import, so references, statuses, and amounts are text and may contain missing or malformed values. Trim whitespace before testing a field. An empty string after trimming can be turned into NULL to give missing data one representation.
SELECT import_id, NULLIF(TRIM(order_ref), '') AS clean_ref
FROM staged_orders
ORDER BY import_id;Compare records at the same grain
The payment ledger may have several charges or refunds for one order. Aggregate that ledger to one row per order before comparing it with shop_orders. Keep unmatched keys visible so a reconciliation reveals missing orders and missing payments.
SELECT order_id,
SUM(CASE WHEN kind = 'charge' THEN amount_cents ELSE -amount_cents END) AS net_cents
FROM payment_ledger
GROUP BY order_id
ORDER BY order_id;Worked solution
SELECT TRIM(order_ref) AS order_ref,MAX(import_id) AS import_id FROM staged_orders WHERE NULLIF(TRIM(order_ref),'') IS NOT NULL GROUP BY TRIM(order_ref) ORDER BY order_ref;Press Control+Space for SQL suggestions. Use the arrow keys to select an option and Enter to insert it.
Results
Loading SQLite…
