快捷導航

Hadoop集成Spring的使用詳細教程(快速入門大數(shù)據(jù))

更新時間：2021年01月22日 09:46:12 作者：瑞新

這篇文章主要介紹了Hadoop集成Spring的使用詳細教程(快速入門大數(shù)據(jù)),本文給大家介紹的非常詳細，對大家的學習或工作具有一定的參考借鑒價值，需要的朋友可以參考下

官網(wǎng)sprng-hadoop

https://spring.io/projects/spring-hadoop

添加依賴

<dependencies>
 <dependency>
  <groupId>org.springframework.data</groupId>
  <artifactId>spring-data-hadoop</artifactId>
  <version>2.5.0.RELEASE</version>
 </dependency>
</dependencies>

使用spring hadoop配置及查看HDFS文件

新建資源文件beans.xml

在這里插入圖片描述

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:hdp="http://www.springframework.org/schema/hadoop"
  xsi:schemaLocation="
  http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
  http://www.springframework.org/schema/hadoop
  http://www.springframework.org/schema/hadoop/spring-hadoop.xsd">

 <hdp:configuration id="hadoopConfiguration">
  fs.defaultFS=hdfs://hadoop01:9000
  hadoop.tmp.dir=/tmp/hadoop
  electric=sea
 </hdp:configuration>

 <hdp:file-system id="fileSystem"
      configuration-ref="hadoopConfiguration"
      user="root"
 />

</beans>

測試文件

package com.bennyrhys.hadoop.spring;

import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.springframework.context.ApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;

import java.io.IOException;

/**
 * @Author bennyrhys
 * @Date 1/21/21 2:35 PM
 */
public class SpringHadoopHDFSApp {
 private ApplicationContext ctx;
 // apache hadoop
 private FileSystem fileSystem;

 /**
  * 創(chuàng)建HDFS文件夾
  */
 @Test
 public void testMkdirs() throws Exception {
  fileSystem.mkdirs(new Path("/springhdfs"));
 }

 /**
  * 查看HDFS文件
  */
 @Test
 public void testText() throws Exception {
  FSDataInputStream in = fileSystem.open(new Path("/springhdfs/hello.txt"));
  IOUtils.copyBytes(in, System.out, 1024);
  in.close();
 }

 @Before
 public void setUp() {
  ctx = new ClassPathXmlApplicationContext("beans.xml");
  fileSystem = (FileSystem) ctx.getBean("fileSystem");
 }

 @After
 public void tearDown() throws Exception {
  ctx = null;
  fileSystem.close();
 }
}

spring hadoop 配置文件詳解

提取變量

使用xml中的頭文件替換bean，使其允許使用上下文
${}導入變量

新建配置文件application.properties

spring.hadoop.fsUri=hdfs://hadoop01:9000

獲取context上下文引入變量
beans.xml

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:hdp="http://www.springframework.org/schema/hadoop"
  xmlns:context="http://www.springframework.org/schema/context"
  xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
  http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
  http://www.springframework.org/schema/hadoop http://www.springframework.org/schema/hadoop/spring-hadoop.xsd">

 <hdp:configuration id="hadoopConfiguration">
  fs.defaultFS=${spring.hadoop.fsUri}
  hadoop.tmp.dir=/tmp/hadoop
  electric=sea
 </hdp:configuration>
 <context:property-placeholder location="application.properties"/>

 <hdp:file-system id="fileSystem"
      configuration-ref="hadoopConfiguration"
      user="root"
 />

</beans>

SpringBoot訪問HDFS系統(tǒng)

pom.xml

<!-- 添加spring boot的依賴操作hadoop -->
 <dependency>
  <groupId>org.springframework.data</groupId>
  <artifactId>spring-data-hadoop-boot</artifactId>
  <version>2.5.0.RELEASE</version>
 </dependency>

SpringBootHDFSApp

package com.bennyrhys.hadoop.spring;

import org.apache.hadoop.fs.FileStatus;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.data.hadoop.fs.FsShell;

/**
 * @Author bennyrhys
 * @Date 1/21/21 11:33 PM
 */
@SpringBootApplication
public class SpringBootHDFSApp implements CommandLineRunner {

 @Autowired
 FsShell fsShell; //引入spring的

 @Override
 public void run(String... strings) throws Exception {
  for (FileStatus fileStatus : fsShell.lsr("/springhdfs")) {
   System.out.println("> " + fileStatus.getPath());
  }
 }

 /**
  * > hdfs://hadoop01:9000/springhdfs
  * > hdfs://hadoop01:9000/springhdfs/hello.txt
  * @param args
  */
 public static void main(String[] args) {
  SpringApplication.run(SpringBootHDFSApp.class, args);
 }
}

到此這篇關(guān)于Hadoop集成Spring的使用詳細教程(快速入門大數(shù)據(jù))的文章就介紹到這了,更多相關(guān)Hadoop集成Spring的使用內(nèi)容請搜索腳本之家以前的文章或繼續(xù)瀏覽下面的相關(guān)文章希望大家以后多多支持腳本之家！

您可能感興趣的文章: