Remove duplicate lines online
How to remove duplicate lines from a list
Duplicate lines creep into every list: email exports merged from two tools, keyword lists from several sources, product SKUs, log lines, URLs from a crawl. This tool scans every line, groups identical ones and returns a clean list with each value once, along with a report of what was removed.
Unlike many tools that sort | uniq your data, it removes duplicate lines without sorting by default. The first occurrence of each line stays in its original position, so ordered lists, rankings and scripts keep their meaning. Choose Keep last occurrence if the newest entry should win, for example when a later row in an export holds updated data.
You can still sort when it helps: A to Z, Z to A, natural order (so item2 comes before item10), by length, by how often each line repeated, reversed or shuffled.
Matching options
Case insensitive, spaces, accents and columns
Two lines that look the same to a person can differ to a computer. Use these options to decide what counts as a duplicate:
- Case insensitive:
Apple, apple and APPLE are treated as one. Essential for emails, domains and tags.
- Ignore extra spaces: leading, trailing and doubled spaces or tabs are ignored. This catches most copy and paste duplicates.
- Ignore accents:
café and cafe match.
- Ignore punctuation:
Hello, world! and hello world match.
- Compare by column: for CSV or TSV data, choose a column number and delimiter. Rows are deduplicated by that column (for example the email column) and the whole row is kept.
The output always shows the line as it was written, not the normalized version used for matching.
Advertisement
Output modes
Find duplicates, unique-only lines and counts
| Output | Input: a, b, a, c | Use it to |
|---|
| Unique lines | a, b, c | Clean a list, keeping one copy of everything |
| Only lines that appear once | b, c | Find values with no match, such as items in one list only |
| Only lines that had duplicates | a | Find the repeated values |
| Unique lines with count | 2 a, 1 b, 1 c | Frequency counts, like sort | uniq -c |
| Keep all, mark duplicates | a [1/2], b, a [2/2], c | Review repeats in context before deleting |
The exclude list removes every line that appears in a second list. Paste unsubscribes, bounced addresses or already-processed IDs there and they are dropped from the result too.
Excel and other tools
Remove duplicate lines in Excel, Notepad++ and the command line
In Excel, select the column and use Data, Remove Duplicates. It is case insensitive and works on whole rows or selected columns, but it changes the sheet in place and does not tell you which values repeated. In Google Sheets, =UNIQUE(A:A) returns unique values in a new range. For a quick check outside a spreadsheet, export the column, paste it here and copy the result back.
Notepad++ can remove consecutive duplicates with Edit, Line Operations, so you usually need to sort first. On the command line, sort -u file.txt or awk '!seen[$0]++' file.txt (which keeps order) do the job for exact matches.
Next steps: fix capitalization with the case converter, compare two versions of a list with text compare, or turn a cleaned CSV into JSON with the CSV to JSON converter.
Examples
Before and after examples
Input with mixed case and spacing: Apple, apple , Banana, apple, Cherry. With Case insensitive and Ignore extra spaces ticked, the result keeps the first spelling of each item in its original position: Apple, Banana, Cherry. Choose Unique lines with count instead and you get 3 Apple, 1 Banana, 1 Cherry, a quick frequency table without a spreadsheet.
FAQ
Remove Duplicate Lines: frequently asked questions
How do I remove duplicate lines?
Paste your list, choose Unique lines as the output and click Remove duplicates. Each line appears once in the result, in its original order.
Can I remove duplicates without sorting?
Yes. Order is set to Keep original order by default, so the first occurrence of every line stays where it was.
Is the comparison case sensitive?
Only if you want it to be. Tick Case insensitive so Apple and apple count as the same line.
How do I remove duplicate rows from a CSV by one column?
Set Compare by column to the column number (for example 2 for the second column) and pick the delimiter. Rows with the same value in that column are removed and the whole first row is kept.
How do I find which lines are duplicated?
Choose Only lines that had duplicates as the output, or check the Duplicated lines table under the result, which lists every repeated line and how many times it appeared.
How do I keep only lines that are not repeated?
Choose Only lines that appear once. Any line that appears two or more times is removed completely.
How many lines can I process?
Up to 5 MB of text, which is hundreds of thousands of lines.
Is my list stored?
No. The list is processed in memory and discarded after the result is shown.