Finding duplicate files on a Mac with the tools already installed

The one-liner, and what it does

find ~/Documents ~/Downloads -type f -size +1M -print0 \
  | xargs -0 shasum -a 256 \
  | sort \
  | awk '{ if ($1 == prev) print; prev = $1 }'

It hashes everything above 1 MB and prints the files whose hash has been seen before. It is correct, and on a large directory it is slow, because it hashes every file rather than only the ones that could possibly match.

The version that finishes

Group by size first, and hash only within groups of two or more. On a 500 GB home folder this is the difference between twenty minutes and twenty hours.

find ~ -type f -size +1M -print0 \
  | xargs -0 stat -f '%z %N' \
  | sort -n \
  | awk '{ s=$1; $1=""; if (s == prev) { if (!shown) print prevline; print substr($0,2); shown=1 } else shown=0; prev=s; prevline=substr($0,2) }' \
  | tr '\n' '\0' \
  | xargs -0 shasum -a 256 \
  | sort \
  | awk '{ if ($1 == prev) print; prev = $1 }'

Three things about this are worth internalising, because they apply to any tool you use instead:

• Size is a perfect filter in one direction. Two files with different sizes are certainly not identical. Two with the same size are only candidates.
• A hash is a claim, not a proof. SHA-256 collisions are not a practical concern, but a truncated or partial hash is a different matter; a tool that hashes only the first 64 KB will call two disk images identical because their headers match.
• Hard links are not duplicates. Two paths to one inode use the space once. Deleting one frees nothing. Compare inode numbers with stat -f '%i' before celebrating.

Never delete both members of a pair, and never let a script delete without printing what it is about to do. The classic accident is a rule that keeps “the first one alphabetically” applied to a list where the original happens to sort second.

What this approach cannot find

Byte-for-byte comparison finds exact copies. It finds nothing at all in the much larger category of near-duplicates:

• The same photo at two resolutions, or exported once for the web and once for print.
• A HEIC original and the JPEG macOS made when you shared it — both on disk, visually identical, not one byte in common. HEIC and JPEG pairs covers why this is now the most common form of photo waste.
• A burst of twelve nearly identical shots, of which you wanted one.
• The same document saved as .docx and as .pdf.

Those need perceptual comparison rather than hashing, which is a different technique with different failure modes — duplicates versus similar photos goes through both.

Where the duplicates actually are

Running this on a whole home folder produces a list dominated by things you must not touch. Three directories generate enormous numbers of legitimate duplicates:

Restrict the search to the last row. Everything found in the first three rows is either structural or actively dangerous to remove, and the rules for deduplicating without losing anything sets out the full list.

Getting the size back

Finding duplicates is the easy half; deciding which copy is the original is the part where people lose files. Three rules that hold up:

1. Keep the copy with the shortest path. It is almost always the one in the place you deliberately put it, rather than a copy that ended up in a download or a temporary folder.
2. Keep the oldest modification time for documents, the newest for exports. A document’s original carries its real history; an export’s newest version is the one you actually produced.
3. Move to the Trash, never rm. On a list of five hundred files you will get one wrong, and the Trash is what makes that recoverable.

Questions

Keep reading