DUPLICATE FILE FINDER LINUX HOW TO

Therefore, sort -u is used to filter duplicate_files and unique_files.

The comm command prints files in duplicate_files but not in unique_files.

Now, we need to remove the files listed in duplicate_files, excluding the files listed in unique_files.

One sample from each group of duplicates is written to unique_files.

We calculate the md5sum value of the duplicates and print one file from each group of duplicates by finding unique lines, comparing md5sum from each line using -w 32 (the first 32 characters in the md5sum output usually, the md5sum output consists of a 32-character hash followed by the filename).

If the checksums of two files are the same, they are confirmed to be duplicates and are printed. The name1 and name2 variables store the consecutive filenames. Here, we read the md5sum checksum of files into the csum1 and csum2 variables.

Once the line is read, the entire line is in $0 and each column is available in $1, $2.

If they are equal, it means that the two files are duplicates by size and must be further checked by md5sum. This block compares the size obtained from the current line and the previously stored size in the size variable. This value will be later stored in RET which is further used to check duplicate fileĮcho $(sha512sum $ block, the rest of the lines are read, one by one. # If file exists and the temporary file's size is zero # Before we check hash value of file, make this variable null # Store the hash of the file if not already added # Find duplicate files using shell script For example /etc/hosts and /etc/hosts.bak are same file so the script will fail to understand which file is important and it may delete /etc/hosts considering it as duplicate.īelow is the sample script to remove duplicate files with a prompt before deleting the file: #!/bin/bash This can help decide the user as which file out of the duplicates he/she wishes to delete. I have modified the script to prompt the user before removing any duplicate file. The script template is taken from bash cookbook.

Script 1: Find duplicate files using shell script In my last article I shared multiple commands and methods to check if the node is connected to internet with shell script in Linux and Unix, Now in this article I will show some sample shell scripts to remove duplicate files. Linux find duplicate files by name and hash value. List the duplicate files in Linux using shell script.

DUPLICATE FILE FINDER LINUX HOW TO

How to find duplicate files using shell script in Linux. Related Searches: How to remove duplicate files in Linux or Unix.

Script 2: Remove duplicate files using shell script.Script 1: Find duplicate files using shell script.

Duplicate file finder linux