# Finding duplicate files on a Mac without installing anything

> Duplicates are found by size, then by hash, then by eye — in that order, because each stage is cheaper than the next. The shell commands that do it, what they miss, and the rules that stop you deleting the wrong copy.

Canonical: https://bytesweep.app/blog/find-duplicate-files-mac.html
Last updated: 2026-09-29

## Finding duplicate files on a Mac with the tools already installed

Every duplicate finder, the expensive ones included, does the same three things in the same order: group files by size, hash only the groups with more than one member, then confirm. The order is about cost. Comparing sizes is free because the file system already knows them. Hashing means reading every byte. Comparing every file against every other file is quadratic and would never finish on a real disk.

- By [Shajeen Ahamed](https://bytesweep.app/about.html#author)
- Published 27 September 2026
- Last updated 29 September 2026
- 4 min read
- Part of [Duplicate file finder](https://bytesweep.app/duplicate-file-finder.html)
- [How we write](https://bytesweep.app/about.html#how-we-write)

You can run that whole pipeline with what macOS ships.

*Each stage is dearer per file than the last, so the cheap filter goes first and the expensive ones see only what survives it.*

### The slow one-liner

```
find ~/Documents ~/Downloads -type f -size +1M -print0 \
  | xargs -0 shasum -a 256 \
  | sort \
  | awk '{ if ($1 == prev) print; prev = $1 }'
```

This hashes everything over 1 MB and prints each file whose hash has already been seen, so the first copy of each set is not printed. It’s correct. It’s also slow on a big folder, because it reads every file, including the thousands that have a unique size and so can’t possibly have a twin.

### The version that finishes

Group by size first and hash only inside groups of two or more:

```
find ~ -type f -size +1M -print0 \
  | xargs -0 stat -f '%z %N' \
  | sort -n \
  | awk '{ s=$1; $1=""; if (s == prev) { if (!shown) print prevline; print substr($0,2); shown=1 } else shown=0; prev=s; prevline=substr($0,2) }' \
  | tr '\n' '\0' \
  | xargs -0 shasum -a 256 \
  | sort \
  | awk '{ if ($1 == prev) print; prev = $1 }'
```

Most files in a home folder have a size nobody else shares, so most files are never opened. On a large disk that is the difference between a run that takes minutes and one you give up on. It matters far more than which hash you pick: reading from disk is the expensive part, and this step removes most of the reading.

Size only filters in one direction. Different sizes prove two files differ; the same size only makes them candidates, which is why the hash still has to run.

The hash itself needs a little care. A SHA-256 collision is not something you’ll ever meet. A partial hash is another matter. Hashing the first and last 64 KB is a perfectly good way to thin out candidates, but a tool that decides on that alone will call two disk images identical because their headers match. Confirm with a full hash, or `cmp`, before anything is deleted.

### Two copies that aren’t using twice the space

A hard link is one file with two names. Both paths point at the same inode, the data is stored once, and deleting one path frees nothing. `stat -f '%i %N' file1 file2` prints the inode numbers; if they match, there’s nothing to gain.

APFS clones are the harder case, because `stat` won’t catch them. A file copied in the Finder on an APFS volume gets its own inode but shares its data blocks with the original until one of them is changed. The hash says duplicate, and it’s right, but deleting either copy returns next to nothing. If a cleanup promises gigabytes and the free space barely moves, clones are the usual explanation.

### What hashing can’t find

Byte comparison finds exact copies and nothing else. It is blind to the near-duplicates, which is often where more of the space is:

- the same photo at two resolutions, or exported once for the web and once for print
- a HEIC original and the JPEG made when you shared it, identical to the eye with not one byte in common ([HEIC and JPEG pairs](https://bytesweep.app/blog/heic-jpeg-duplicates.html) explains where those come from)
- a burst of twelve shots of which you wanted one
- the same document saved as `.docx` and as `.pdf`

Those need perceptual comparison, a different technique with different ways of going wrong. [Duplicate versus similar photos](https://bytesweep.app/blog/duplicate-photos-vs-similar.html) covers it.

### Point it at the right folders

Run the pipeline over your whole home folder and the output is dominated by things you must not touch:

| Directory | Why it is full of duplicates | Delete? |
| --- | --- | --- |
| `node_modules`, `.venv`, `vendor` | Every project vendors the same libraries | No. Delete the whole folder instead; it reinstalls |
| `~/Library/Application Support` | Applications keep versioned copies of their own data | No |
| Photos and Lightroom libraries | Managed libraries keep originals plus renditions | No. Never reach inside a library |
| `~/Downloads`, `~/Desktop`, `~/Documents` | Genuine human duplication | Yes, this is the useful set |

Restrict the search to the last row. The other three are either structural or actively dangerous to prune file by file. A Photos library in particular should only be deduplicated from inside Photos, with its Duplicates album, because the files on disk are storage for a database and not a folder of pictures. The rules for deduplicating without losing anything has the full exclusion list.

### Choosing which copy goes

Finding duplicates is the easy half. Deciding which copy is the original is where people lose files. I’d use these three rules and not improvise:

1. Keep the copy with the shortest path. It’s nearly always the one in the place you put it on purpose, not the one that landed in a download or a temporary folder.
2. For documents, keep the oldest modification time, because the original carries the real history. For exports, keep the newest, because that’s the one you actually produced.
3. Move to the Trash, never `rm`. On a list of five hundred files you will get one wrong, and the Trash is what makes that recoverable.

Never delete both members of a pair, and never let a script delete anything without first printing what it is about to do. The classic accident is a rule that keeps “the first one alphabetically”, applied to a list where the original happens to sort second.

### Let Bytesweep clear this for you

Skip the commands. Bytesweep finds this for you, rates every item safe, review first or keep, and clears what you choose. Most of what it removes can be put back for seven days. See what it finds on your own disk, free for 3 days, then pay once with no subscription.

[Try it free for 3 days](https://bytesweep.app/download.html) [Buy · $29.99](https://bytesweep.app/pricing.html)

### Keep reading

- [**Duplicate photos and similar photos are different problems** A hash finds exact copies and nothing else. Perceptual comparison finds the burst shots, re-saves and resized exports that actually fill a photo library, with a false-positive rate you have to manage.](https://bytesweep.app/blog/duplicate-photos-vs-similar.html)
- [**HEIC and JPEG pairs are quietly doubling your photo library** macOS and iOS convert HEIC to JPEG whenever you share, export or edit in an app that cannot read HEIC, and keep both. Where the copies come from, how to find them, and which one to keep.](https://bytesweep.app/blog/heic-jpeg-duplicates.html)
- [**Inside a Photos library: originals, renditions and the database** A Photos library is a database with a file store attached, and most of its size is not your photographs. What each folder holds, why the library is larger than the sum of your pictures, and what you must never delete.](https://bytesweep.app/blog/photos-library-size.html)