Remove Duplicate Lines in Bash

Deduplicating lines is a common text-processing task. The right tool depends on whether you care about preserving the original line order.

Quick-and-dirty: sort -u

cat $FILE | sort -u

Fast and simple, but sorts the output lexicographically. Use when order doesn’t matter.

Preserve first occurrence (order-stable)

The canonical awk one-liner:

awk '!seen[$0]++' $FILE

seen is an associative array keyed by the entire line. The first time a line is seen, seen[$0] is 0 (false), so the line prints and the counter increments. Subsequent occurrences evaluate to 1 (true) and are suppressed.

Preserve last occurrence

If you want to keep the last occurrence of each line while maintaining relative order:

tac $FILE | awk '!seen[$0]++' | tac

tac reverses line order, so the awk filter sees the last occurrence first, then reverses back.

Complex alternative: sort by line number

For pipelines where awk is unavailable:

cat $FILE | cat -n | sort -uk2 | sort -nk1 | cut -f2-

This numbers each line, sorts by content (keeping only unique lines), re-sorts by the original line number, and strips the numbering.

Sources

• GNU Awk User’s Guide — Arrays
• Stack Overflow — Remove duplicate lines without sorting

Related: bash-scripting, debugging-bash-scripts-set-x