Line Dedupe & Sort

Line dedupe and sort: the boundaries of exact matching and common surprises

Dedupe is "exact match", not "looks the same"

To decide whether two lines are duplicates, the tool does a character-by-character comparison. So these "look-alike" cases are not removed:

So the most common reason "dedupe did not work" is trailing whitespace or invisible characters. Text copied from a web page or a PDF is especially likely to carry them. The recommended order is: remove blank lines and trim first, then dedupe. This tool offers a "remove blank lines" toggle and compares by exact line content when judging duplicates.

Also note: a field that itself contains a newline is not "one line". When copying from CSV, if a field has an internal newline (and was not quoted), it is split into multiple lines, and dedupe then works at the wrong granularity.

Which one is kept

On a duplicate, the tool keeps the first occurrence and discards the later ones. If the surrounding context carried position information, the deduped result loses it.

When you need "keep the last" or "pick one by a rule", line dedupe is insufficient — that is data dedupe, which groups by a key field and takes one row, using SQL's DISTINCT ON or row_number(), or a script.

Two notions of sorting

The tool can sort alphabetically, and here too there are details:

• Code-point sort (the default string comparison). Ordered by the Unicode value of characters, with the result that all uppercase letters come before lowercase: Z precedes a.
• Case-insensitive sort. Letters interleave more intuitively, but the result depends on language rules (for example German ö versus o, or Swedish placing å after z).
• Numeric sort. Under string comparison 10 precedes 2 (because the first character 1 is less than 2). A natural order (2 before 10) needs a numeric comparison.

When you see "the sort is wrong", first decide which kind it is. The dedupe-plus-sort combination is most often used to organize keywords, domains, emails, IPs, and import whitelists — here the "duplicates" are usually format differences (one with www, one without; one with a trailing slash, one without), and you must normalize before dedupe or it will not clean up fully.

Actual uses of dedupe

• Merge multiple data sources. Concatenate several exported lists, then dedupe.
• Clean crawl results. The same link appears many times in scraper output.
• Verify differences. Dedupe and sort both lists, then use the text-diff tool to find differences — far more reliable than manual checking. These two steps used together are a very practical combination.
• Compact config files. When merging several environment-variable files, remove duplicates (but first confirm same-name-different-value items — those are conflicts and should not be silently deduped).

The last point deserves emphasis: dedupe hides conflicts. Two rows with the same name but different values are not "duplicates"; after dedupe you lose one of them and will not know which. When merging config, find the same-name-different-value items and confirm them by hand before doing any pure deduplication.

How to use the tool

Paste text with one item per line, toggle "remove blank lines", "dedupe", and "sort" as needed, click Process, then use the Copy button to take the result. The three toggles combine freely — for example, to remove only blank lines while keeping duplicates, uncheck dedupe. The whole operation happens in the page; nothing is uploaded.