标签：rem ready ash pre cep style form you int
How would you remove duplicate lines from a file that is much too large to fit in memory? The duplicate lines are not necessarily adjacent, and say the file is 10 times bigger than RAM.
A better solution is to use HashSet to store each line of input.txt. As set ignores duplicate values, so while storing a line, check if it already present in hashset. Write it to output.txt only if not present in hashset.
Java:
// Efficient Java program to remove
// duplicates from input.txt and
// save output to output.txt
import java.io.*;
import java.util.HashSet;
public class FileOperation
{
public static void main(String[] args) throws IOException
{
// PrintWriter object for output.txt
PrintWriter pw = new PrintWriter("output.txt");
// BufferedReader object for input.txt
BufferedReader br = new BufferedReader(new FileReader("input.txt"));
String line = br.readLine();
// set store unique values
HashSet<String> hs = new HashSet<String>();
// loop for each line of input.txt
while(line != null)
{
// write only if not
// present in hashset
if(hs.add(line))
pw.println(line);
line = br.readLine();
}
pw.flush();
// closing resources
br.close();
pw.close();
System.out.println("File operation performed successfully");
}
}
How to remove duplicate lines in a large text file?
标签：rem ready ash pre cep style form you int
原文地址：https://www.cnblogs.com/lightwindy/p/9650718.html
