CSV Duplicate Remover by Selected Keys
CSV duplicate remover finds records whose selected key columns match and keeps the first or last record you choose.
Comma-delimited UTF-8 CSV only; up to 100,000 characters, 1,000 records including header and 100 columns. Files are read in this page; no file upload or account is needed. This page loads analytics scripts, so avoid confidential source text.
Output preserves parsed cell text, not original quoting, BOM or record separators. Blank physical records are padded to first-record width; ragged explicit rows and malformed quotes stop export. See RFC 4180 format guidance.
CSV export preserves cell text; it does not make formula-like cells safe for spreadsheet software. Review cells before opening in a spreadsheet. OWASP CSV Injection guidance.
Next: select and reorder CSV columns, convert deduplicated CSV to JSON or remove duplicate text lines (physical lines, not parsed CSV records).
Why deduplicate CSV by selected keys?
Define a duplicate
Choose one field, several fields or whole record. Other columns may differ without preventing a selected-key match, which is what lets an updated note or a new timestamp sit beside the same order number.
Choose survivor
Keep first occurrence or last occurrence, including its non-key values. The surviving row is copied whole; nothing is stitched together from two records.
See the effect
Review source, distinct-key, kept and removed counts before copying or downloading, so a key that groups too much is obvious before you export.
How CSV duplicate removal works
The tool compares parsed records, not physical text lines. A quoted cell may contain a newline without starting another record. Selected column values form one composite key, and records with the same key belong to a duplicate group, compared exactly as parsed.
If three rows have keys 7, 8 and 7, keep-first retains the first 7 and 8. Keep-last retains 8 and the final 7, in the order those rows appear in the source file. Non-key cells come from the chosen survivor, not a merge, so a later correction is either kept entirely or dropped entirely.
Comparison and output rules
- Exact valuesCase, spaces and punctuation are significant. Empty cells can match other empty cells, so blank keys group together and the survivor rule picks which one remains.
- Composite keysAll selected positions must match. Column position distinguishes duplicate or blank headers, so two id columns never collapse into one key.
- Whole recordsOutput retains complete surviving rows and a single header when header mode is enabled. Selecting every column makes only fully repeated rows duplicates.
- Survivor orderKeep-first and keep-last both emit surviving rows in original source order. The difference is which member of each duplicate group survives, and which values travel with it.
- Header handlingWith header on, record one is never compared as data and is written back once. With header off, every record is data, so a first row that repeats later text can be treated as a duplicate.
- CountsSource counts data records only. Distinct keys counts groups, kept counts survivors, and removed is source minus kept, so the three numbers always reconcile.
Common Use Cases
Latest export wins
Keep last record for an order key when later rows carry an update, provided the export appends newer rows rather than sorting them.
Earliest registration wins
Keep first occurrence and remove later repeats for a chosen identifier, which preserves the signup that actually arrived first.
Remove exact repeated records
Select every column when only an entirely repeated row counts as duplicate, leaving rows that differ in any single cell untouched.
How to remove duplicate CSV rows
Add valid CSV
Paste or open comma-separated UTF-8 data and confirm header mode.
Read columns and choose keys
Read the columns, then select one or more positions, or all columns for whole-record matching.
Choose first or last
Run removal and inspect complete surviving records and counts.
Export current CSV
Copy or download only after result matches intended survivor policy.
Controlled record cleanup
Parsed multiline records
Quoted newlines do not create false duplicate rows, so a two-line address is compared as one value rather than two records.
No silent merging
Values from different survivors are not combined into a new record. What you keep is a row that existed in the source.
Recoverable errors
Malformed input disables export and leaves source available for correction, so a failed run never costs the pasted data.
Before deleting duplicates
- •Choose keys carefully: A broad key such as surname can group records that are not the same entity. Start narrow and check the removed count before widening.
- •Expect exact matching: Different case or surrounding spaces remain different in this version, so trim in the source first if you expected a match.
- •Review spreadsheet risk: Cell values stay unchanged; formula-like content needs review before opening elsewhere in a spreadsheet.
- •Check survivor policy: Switch between first and last on the same data and compare which row survives for one key before exporting a large file.
Deduplicate parsed CSV records
Add CSV above, choose key positions and select first or last survivor. Blank keys are compared like any other value, so decide whether those rows belong in the result.
Review counts and complete rows before exporting current result.
Related tools
Frequently Asked Questions
What is the difference between keep first and keep last?
Keep first retains the earliest record for each key. Keep last retains the final occurrence and preserves the original order of surviving records.
Can I match more than one column?
Yes. Records are duplicates only when every selected column matches exactly. Select all columns for whole-record comparison.
Does it ignore letter case or trim spaces?
No. Comparison uses exact parsed cell values. Case, spaces and punctuation remain significant.
How are blank or duplicate headers handled?
Select columns by one-based position, not header name. Blank header is still a position, and repeated names remain separate fields.