Java訪問Hadoop分布式文件系統(tǒng)HDFS的配置說明
更新時(shí)間：2016年06月02日 08:58:35 作者：zhangzhaokun
Hadoop的能提供高吞吐量的數(shù)據(jù)訪問,是集群式服務(wù)器的上的數(shù)據(jù)操作利器,這里就來為大家分享Java訪問Hadoop分布式文件系統(tǒng)HDFS的配置說明:
配置文件
m103替換為hdfs服務(wù)地址。
要利用Java客戶端來存取HDFS上的文件，不得不說的是配置文件hadoop-0.20.2/conf/core-site.xml了，最初我就是在這里吃了大虧，所以我死活連不上HDFS，文件無法創(chuàng)建、讀取。
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <!--- global properties --> <property> <name>hadoop.tmp.dir</name> <value>/home/zhangzk/hadoop</value> <description>A base for other temporary directories.</description> </property> <!-- file system properties --> <property> <name>fs.default.name</name> <value>hdfs://linux-zzk-113:9000</value> </property> </configuration>
配置項(xiàng)：hadoop.tmp.dir表示命名節(jié)點(diǎn)上存放元數(shù)據(jù)的目錄位置，對于數(shù)據(jù)節(jié)點(diǎn)則為該節(jié)點(diǎn)上存放文件數(shù)據(jù)的目錄。
配置項(xiàng)：fs.default.name表示命名的IP地址和端口號(hào),缺省值是file:///，對于JavaAPI來講，連接HDFS必須使用這里的配置的URL地址，對于數(shù)據(jù)節(jié)點(diǎn)來講，數(shù)據(jù)節(jié)點(diǎn)通過該URL來訪問命名節(jié)點(diǎn)。
hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?> <!--Autogenerated by Cloudera Manager--> <configuration> <property> <name>dfs.namenode.name.dir</name> <value>file:///mnt/sdc1/dfs/nn</value> </property> <property> <name>dfs.namenode.servicerpc-address</name> <value>m103:8022</value> </property> <property> <name>dfs.https.address</name> <value>m103:50470</value> </property> <property> <name>dfs.https.port</name> <value>50470</value> </property> <property> <name>dfs.namenode.http-address</name> <value>m103:50070</value> </property> <property> <name>dfs.replication</name> <value>3</value> </property> <property> <name>dfs.blocksize</name> <value>134217728</value> </property> <property> <name>dfs.client.use.datanode.hostname</name> <value>false</value> </property> <property> <name>fs.permissions.umask-mode</name> <value>022</value> </property> <property> <name>dfs.namenode.acls.enabled</name> <value>false</value> </property> <property> <name>dfs.block.local-path-access.user</name> <value>cloudera-scm</value> </property> <property> <name>dfs.client.read.shortcircuit</name> <value>false</value> </property> <property> <name>dfs.domain.socket.path</name> <value>/var/run/hdfs-sockets/dn</value> </property> <property> <name>dfs.client.read.shortcircuit.skip.checksum</name> <value>false</value> </property> <property> <name>dfs.client.domain.socket.data.traffic</name> <value>false</value> </property> <property> <name>dfs.datanode.hdfs-blocks-metadata.enabled</name> <value>true</value> </property> <property> <name>fs.http.impl</name> <value>com.scistor.datavision.fs.HTTPFileSystem</value> </property> </configuration>
mapred-site.xml
<?xml version="1.0" encoding="UTF-8"?> <!--Autogenerated by Cloudera Manager--> <configuration> <property> <name>mapreduce.job.split.metainfo.maxsize</name> <value>10000000</value> </property> <property> <name>mapreduce.job.counters.max</name> <value>120</value> </property> <property> <name>mapreduce.output.fileoutputformat.compress</name> <value>true</value> </property> <property> <name>mapreduce.output.fileoutputformat.compress.type</name> <value>BLOCK</value> </property> <property> <name>mapreduce.output.fileoutputformat.compress.codec</name> <value>org.apache.hadoop.io.compress.SnappyCodec</value> </property> <property> <name>mapreduce.map.output.compress.codec</name> <value>org.apache.hadoop.io.compress.SnappyCodec</value> </property> <property> <name>mapreduce.map.output.compress</name> <value>true</value> </property> <property> <name>zlib.compress.level</name> <value>DEFAULT_COMPRESSION</value> </property> <property> <name>mapreduce.task.io.sort.factor</name> <value>64</value> </property> <property> <name>mapreduce.map.sort.spill.percent</name> <value>0.8</value> </property> <property> <name>mapreduce.reduce.shuffle.parallelcopies</name> <value>10</value> </property> <property> <name>mapreduce.task.timeout</name> <value>600000</value> </property> <property> <name>mapreduce.client.submit.file.replication</name> <value>1</value> </property> <property> <name>mapreduce.job.reduces</name> <value>24</value> </property> <property> <name>mapreduce.task.io.sort.mb</name> <value>256</value> </property> <property> <name>mapreduce.map.speculative</name> <value>false</value> </property> <property> <name>mapreduce.reduce.speculative</name> <value>false</value> </property> <property> <name>mapreduce.job.reduce.slowstart.completedmaps</name> <value>0.8</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>m103:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>m103:19888</value> </property> <property> <name>mapreduce.jobhistory.webapp.https.address</name> <value>m103:19890</value> </property> <property> <name>mapreduce.jobhistory.admin.address</name> <value>m103:10033</value> </property> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>yarn.app.mapreduce.am.staging-dir</name> <value>/user</value> </property> <property> <name>mapreduce.am.max-attempts</name> <value>2</value> </property> <property> <name>yarn.app.mapreduce.am.resource.mb</name> <value>2048</value> </property> <property> <name>yarn.app.mapreduce.am.resource.cpu-vcores</name> <value>1</value> </property> <property> <name>mapreduce.job.ubertask.enable</name> <value>false</value> </property> <property> <name>yarn.app.mapreduce.am.command-opts</name> <value>-Djava.net.preferIPv4Stack=true -Xmx1717986918</value> </property> <property> <name>mapreduce.map.java.opts</name> <value>-Djava.net.preferIPv4Stack=true -Xmx1717986918</value> </property> <property> <name>mapreduce.reduce.java.opts</name> <value>-Djava.net.preferIPv4Stack=true -Xmx2576980378</value> </property> <property> <name>yarn.app.mapreduce.am.admin.user.env</name> <value>LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH</value> </property> <property> <name>mapreduce.map.memory.mb</name> <value>2048</value> </property> <property> <name>mapreduce.map.cpu.vcores</name> <value>1</value> </property> <property> <name>mapreduce.reduce.memory.mb</name> <value>3072</value> </property> <property> <name>mapreduce.reduce.cpu.vcores</name> <value>1</value> </property> <property> <name>mapreduce.application.classpath</name> <value>$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,$MR2_CLASSPATH,$CDH_HCAT_HOME/share/hcatalog/*,$CDH_HIVE_HOME/lib/*,/etc/hive/conf,/opt/cloudera/parcels/CDH/lib/udps/*</value> </property> <property> <name>mapreduce.admin.user.env</name> <value>LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH</value> </property> <property> <name>mapreduce.shuffle.max.connections</name> <value>80</value> </property> </configuration>
利用JavaAPI來訪問HDFS的文件與目錄
package com.demo.hdfs;
import java.io.BufferedInputStream;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URI;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.util.Progressable;
/**
* @author zhangzk
*
*/
public class FileCopyToHdfs {
public static void main(String[] args) throws Exception {
try {
//uploadToHdfs();
//deleteFromHdfs();
//getDirectoryFromHdfs();
appendToHdfs();
readFromHdfs();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
finally
{
System.out.println("SUCCESS");
}
}
/**上傳文件到HDFS上去*/
private static void uploadToHdfs() throws FileNotFoundException,IOException {
String localSrc = "d://qq.txt";
String dst = "hdfs://192.168.0.113:9000/user/zhangzk/qq.txt";
InputStream in = new BufferedInputStream(new FileInputStream(localSrc));
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(URI.create(dst), conf);
OutputStream out = fs.create(new Path(dst), new Progressable() {
public void progress() {
System.out.print(".");
}
});
IOUtils.copyBytes(in, out, 4096, true);
}
/**從HDFS上讀取文件*/
private static void readFromHdfs() throws FileNotFoundException,IOException {
String dst = "hdfs://192.168.0.113:9000/user/zhangzk/qq.txt";
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(URI.create(dst), conf);
FSDataInputStream hdfsInStream = fs.open(new Path(dst));
OutputStream out = new FileOutputStream("d:/qq-hdfs.txt");
byte[] ioBuffer = new byte[1024];
int readLen = hdfsInStream.read(ioBuffer);
while(-1 != readLen){
out.write(ioBuffer, 0, readLen);
readLen = hdfsInStream.read(ioBuffer);
}
out.close();
hdfsInStream.close();
fs.close();
}
/**以append方式將內(nèi)容添加到HDFS上文件的末尾;注意：文件更新，需要在hdfs-site.xml中添<property><name>dfs.append.support</name><value>true</value></property>*/
private static void appendToHdfs() throws FileNotFoundException,IOException {
String dst = "hdfs://192.168.0.113:9000/user/zhangzk/qq.txt";
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(URI.create(dst), conf);
FSDataOutputStream out = fs.append(new Path(dst));
int readLen = "zhangzk add by hdfs java api".getBytes().length;
while(-1 != readLen){
out.write("zhangzk add by hdfs java api".getBytes(), 0, readLen);
}
out.close();
fs.close();
}
/**從HDFS上刪除文件*/
private static void deleteFromHdfs() throws FileNotFoundException,IOException {
String dst = "hdfs://192.168.0.113:9000/user/zhangzk/qq-bak.txt";
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(URI.create(dst), conf);
fs.deleteOnExit(new Path(dst));
fs.close();
}
/**遍歷HDFS上的文件和目錄*/
private static void getDirectoryFromHdfs() throws FileNotFoundException,IOException {
String dst = "hdfs://192.168.0.113:9000/user/zhangzk";
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(URI.create(dst), conf);
FileStatus fileList[] = fs.listStatus(new Path(dst));
int size = fileList.length;
for(int i = 0; i < size; i++){
System.out.println("name:" + fileList[i].getPath().getName() + "/t/tsize:" + fileList[i].getLen());
}
fs.close();
}
}
注意：對于append操作，從hadoop-0.21版本開始就不支持了，關(guān)于Append的操作可以參考Javaeye上的一篇文檔。
相關(guān)文章
Spring配置多個(gè)數(shù)據(jù)源并實(shí)現(xiàn)動(dòng)態(tài)切換示例
本篇文章主要介紹了Spring配置多個(gè)數(shù)據(jù)源并實(shí)現(xiàn)動(dòng)態(tài)切換示例，小編覺得挺不錯(cuò)的，現(xiàn)在分享給大家，也給大家做個(gè)參考。一起跟隨小編過來看看吧2017-04-04
spring boot集成shiro詳細(xì)教程(小結(jié))
這篇文章主要介紹了spring boot 集成shiro詳細(xì)教程，小編覺得挺不錯(cuò)的，現(xiàn)在分享給大家，也給大家做個(gè)參考。一起跟隨小編過來看看吧2018-01-01
Netty分布式抽象編碼器MessageToByteEncoder邏輯分析
這篇文章主要介紹了Netty分布式抽象編碼器MessageToByteEncoder的抽象邏輯分析，有需要的朋友可以借鑒參考下，希望能夠有所幫助，祝大家多多進(jìn)步，早日升職加薪2022-03-03
這篇文章主要介紹了Java中四種字符串替換方法,分別是String.replace()、String.replaceAll()、String.replaceFirst()和StringBuilder.replace(),并對比了它們的特點(diǎn)和適用場景,需要的朋友可以參考下2025-02-02
javaweb中Filter(過濾器)的常見應(yīng)用
這篇文章主要介紹了javaweb中Filter的常見應(yīng)用，過濾器的使用方法，感興趣的小伙伴們可以參考一下2015-12-12
javaweb如何實(shí)現(xiàn)請求和響應(yīng)
這篇文章主要為大家詳細(xì)介紹了javaweb如何實(shí)現(xiàn)請求和響應(yīng),具有一定的參考價(jià)值，感興趣的小伙伴們可以參考一下2016-10-10
SpringBoot如何優(yōu)雅實(shí)現(xiàn)接口參數(shù)驗(yàn)證
為了保證參數(shù)的正確性,我們需要使用參數(shù)驗(yàn)證機(jī)制,來檢測并處理傳入的參數(shù)格式是否符合規(guī)范,所以本文就來和大家聊聊如何優(yōu)雅實(shí)現(xiàn)接口參數(shù)驗(yàn)證吧2023-08-08
命令模式是一種行為設(shè)計(jì)模式,它允許您將請求封裝為對象,以便您可以將其參數(shù)化、隊(duì)列化、記錄和撤銷,本文主要為大家介紹一下java實(shí)現(xiàn)命令模式的示例代碼,需要的可以參考下2023-09-09
Java報(bào)錯(cuò)：FileNotFoundException的解決方案
在Java編程中,FileNotFoundException 是一種常見的受檢異常,通常發(fā)生在試圖打開一個(gè)不存在的文件或文件路徑錯(cuò)誤時(shí),本文將詳細(xì)探討FileNotFoundException的成因、解決方案以及預(yù)防措施,幫助開發(fā)者理解和避免此類問題,需要的朋友可以參考下2024-06-06
這篇文章主要介紹了springcloud集成zookeeper的方法示例，小編覺得挺不錯(cuò)的，現(xiàn)在分享給大家，也給大家做個(gè)參考。一起跟隨小編過來看看吧2018-04-04