标签：false tle gre amazon s3 相关 rename buffere 移动文件 命令
HDFS，是Hadoop Distributed File System的简称，是Hadoop抽象文件系统的一种实现。Hadoop抽象文件系统可以与本地系统、Amazon S3等集成，甚至可以通过Web协议（webhsfs）来操作。HDFS的文件分布在集群机器上，同时提供副本进行容错及可靠性保证。例如客户端写入读取文件的直接操作都是分布在集群各个机器上的，没有单点性能压力。
HDFS相关的搭建可以看我前面的一篇博文，我们今天主要来讲下怎么操作hdfs的api和 hdfs命令行，
<repositories>
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
</repository>
</repositories>
//导包
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoop.version}</version>
</dependency>
this.configuration = new Configuration();
this.fileSystem = FileSystem.get(new URI(this.HDFS_PATH),configuration,"hadoop");
Path path = new Path("/hdfsapi/test");
boolean result = fileSystem.mkdirs(path);
Path path = new Path("/gwyy.txt");
FSDataInputStream fsDataInputStream = fileSystem.open(path);
FileOutputStream fileOutputStream = new FileOutputStream(new File("a.txt"));
byte[] buffer = new byte[1024];
int length = 0;
StringBuffer sb = new StringBuffer();
while( ( length = fsDataInputStream.read(buffer)) != -1) {
sb.append(new String(buffer,0,buffer.length));
fileOutputStream.write(buffer,0,buffer.length);
}
System.out.println(sb.toString());
FSDataOutputStream out = fileSystem.create(new Path("/fuck.txt"));
out.writeUTF("aaabbb");
out.flush();
out.close();
boolean a = fileSystem.rename(new Path("/fuck.txt"),new Path("/fuck.aaa"));
System.out.println(a);
fileSystem.copyFromLocalFile(new Path("a.txt"),new Path("/copy_a.txt"));
InputStream in = new BufferedInputStream(new FileInputStream(new File("hive-1.1.0-cdh5.15.1.tar.gz")));
Path dst = new Path("/hive.tar.gz");
//显示进度条
FSDataOutputStream out = fileSystem.create(dst, new Progressable() {
@Override
public void progress() {
System.out.flush();
System.out.print(‘.‘);
}
});
byte[] buffer = new byte[4096];
int length = 0;
//写入到 hdfs
while((length = in.read(buffer,0,buffer.length)) != -1) {
out.write(buffer,0,buffer.length);
}
fileSystem.copyToLocalFile(new Path("/fuck.aaa"),new Path("./"));
FileStatus[] fileStatuses = fileSystem.listStatus(new Path("/"));
for (FileStatus f:fileStatuses) {
System.out.println(f.getPath());
}
RemoteIterator<LocatedFileStatus> remoteIterator = fileSystem.listFiles(new Path("/"),true);
while(remoteIterator.hasNext()) {
LocatedFileStatus file = remoteIterator.next();
System.out.println(file.getPath());
}
FileStatus fileStatus = fileSystem.getFileStatus(new Path("/jdk-8u221-linux-x64.tar.gz"));
BlockLocation[] blockLocations = fileSystem.getFileBlockLocations(fileStatus,0,fileStatus.getLen());
//查看区块
for (BlockLocation b:blockLocations) {
for (String name:b.getNames()) {
System.out.println(name + b.getOffset() + b.getLength());
}
}
如果路径是目录并设置为*如果为true，则删除目录，否则引发异常。在*对于文件，递归可以设置为true或false。
boolean a = fileSystem.delete(new Path("/gwyy.txt"),true);
System.out.println(a);
hadoop fs -ls /
hadoop fs -put gwyy.txt /
hf -copyFromLocal xhc.txt /
####从本地移动文件到hdfs 本地文件删除 hf -moveFromLocal a.txt /
hadoop fs -cat /gwyy.txt hadoop fs -text /gwyy.txt
hadoop fs -get /a.txt ./
hadoop fs -mkdir /hdfs-test
hadoop fs -mv /a.txt /hdfs-test/a.txt
hadoop fs -cp /hdfs-test/a.txt /hdfs-test/a.txt.back
hadoop fs -getmerge /hdfs-test ./t.txt
hf -rm /hdfs-test/a.txt.back
hadoop fs -rmdir /hdfs-test 只能删除空目录 hadoop fs -rm -r /hdfs-test 删除目录不管有没有东西都删
标签：false tle gre amazon s3 相关 rename buffere 移动文件 命令
原文地址：https://www.cnblogs.com/gwyy/p/12205199.html
