标签:
InputStreamReader isr = new InputStreamReader(new FileInputStream("./test/垃圾短信训练集80W条.txt"),"UTF-8");
//FileReader fr = new FileReader(new File("./test/垃圾短信训练集80W条.txt"));
BufferedReader br = new BufferedReader(isr);
//FileWriter fw = new FileWriter(new File("./test/垃圾短信训练集80W条_result.txt"), true);
OutputStreamWriter osw = new OutputStreamWriter(new FileOutputStream("./test/垃圾短信训练集80W条_result.txt"),"UTF-8");
BufferedWriter bw = new BufferedWriter(osw);
问题出在FileReader读取文件的过程中,FileReader继承了InputStreamReader,但并没有实现父类中带字符集参数的构造函数,
所以FileReader只能按系统默认的字符集来解码,然后在UTF-8 -> GBK -> UTF-8的过程中编码出现损失,造成结果不能还原最初的字符。
原因明确了,这个问题解决起来并不困难,用InputStreamReader代替FileReader,
InputStreamReader isr=new InputStreamReader(new FileInputStream(fileName),"UTF-8");
这样读取文件就会直接用UTF-8解码,不用再做编码转换。
标签:
原文地址:http://www.cnblogs.com/hlongch/p/5837849.html