Remove Duplicate Lines
Processed Client SideRemove duplicate lines from any text or list, with case, trim, sort, and keep-first/last options.
Bookmark this tool now — skip the search next time you need it.
About Remove Duplicate Lines
This tool runs entirely in your browser. Whatever you paste is processed on your own device and is never uploaded, logged, or sent to any server.
Remove duplicate lines from any list or block of text, with enough control to handle the messy real-world cases rather than just the clean ones. You can ignore case so that Apple and apple count as the same entry, trim surrounding whitespace so that trailing spaces stop hiding duplicates, drop empty lines entirely, and choose whether the first or the last occurrence of each duplicate survives. The result can keep the original order or be sorted alphabetically in either direction. It is the fastest way to clean an exported list of emails, tidy a log extract, or deduplicate a set of IDs before pasting them into a query. The output reports how many lines went in, how many survived, and how many were removed, so a deduplication that removes far more than you expected is visible immediately rather than after the fact.
Key features
- Removes duplicate lines while preserving the original order by default
- Ignore case, so entries differing only in capitalisation are treated as duplicates
- Trim lines, so leading and trailing whitespace stops concealing duplicates
- Drop empty lines to clear out the blank rows an export leaves behind
- Keep first or keep last, controlling which occurrence of each duplicate survives
- Optional alphabetical sort, ascending or descending
- Counts for lines in, lines kept, and lines removed, so an unexpectedly aggressive run is obvious straight away
- Comparison and output kept separate — ignoring case or trimming affects what counts as a duplicate without rewriting the lines you keep
- Handles long lists without a size limit or a server round-trip
- Runs entirely in your browser — email lists and exports are never uploaded
How to use it
- Paste your lines into the input pane.
- Turn on Trim lines and Ignore case if your data is inconsistently formatted.
- Turn on Drop empty to remove blank lines.
- Choose Keep first or Keep last, and a sort order if you want one.
- Click Remove Duplicates and copy the result.
Tips & common mistakes
- Turn Trim lines on almost always. Invisible trailing whitespace is the most common reason two apparently identical lines are not treated as duplicates.
- Keep first preserves the earliest occurrence, which is what you want for a chronological log. Keep last preserves the most recent, which is what you want when later entries supersede earlier ones.
- Ignore case is right for emails and domain names, which are case-insensitive in practice, and wrong for identifiers, tokens, and anything case-sensitive.
- Sorting makes near-duplicates visible by putting them next to each other — useful for spotting entries that differ by a typo rather than exactly.
- For deduplicating structured records rather than lines, convert to JSON first; deduplicating CSV rows as text will treat any column difference as a distinct row.
- Everything runs locally, so this is safe for customer email lists and other exports containing personal data.
- Keep last also moves the surviving line to the position of its last occurrence, so the output order changes as well as which copy survives. Keep first is the option that leaves the order of the remaining lines untouched.
- Sorting is alphabetical by locale rather than by character code, so accented letters sort next to their unaccented forms instead of after every plain letter. That is usually what you want for names, and worth knowing if you are comparing the result against a database ORDER BY.