java spark文件讀取亂碼問題的解決方法

更新時間：2024年10月28日 10:35:51 作者：cah黎明就在眼前

這篇文章主要為大家詳細介紹了java spark文件讀取亂碼問題的相關解決方法,文中的示例代碼講解詳細,感興趣的小伙伴可以跟隨小編一起學習一下

一、問題

環(huán)境為jdk1.8，spark3.2.1，讀取hadoop中GB18030編碼格式的文件出現(xiàn)亂碼。

二、心酸歷程

為了解決該問題，嘗試過很多種方法，但都沒有成功

1、textFile+Configuration方式——亂碼

        String filePath = "hdfs:///user/test.deflate";
        //創(chuàng)建SparkSession和SparkContext的實例
        String encoding = "GB18030";

        SparkSession spark = SparkSession.builder()
                .master("local[*]").appName("Spark Example")
                .getOrCreate();

        JavaSparkContext sc = JavaSparkContext.fromSparkContext(spark.sparkContext());
        Configuration entries = sc.hadoopConfiguration();
        entries.set("textinputformat.record.delimiter", "\n");
        entries.set("mapreduce.input.fileinputformat.inputdir",filePath);entries.set("mapreduce.input.fileinputformat.encoding", "GB18030");
        
        JavaRDD<String> rdd = sc.textFile(filePath);

2、spark.read().option方式——亂碼

        Dataset<Row> load = spark.read().format("text").option("encoding", "GB18030").load(filePath);

        load.foreach(row -> {
            System.out.println(row.toString());
            System.out.println(new String(row.toString().getBytes(encoding),"UTF-8"));
            System.out.println(new String(row.toString().getBytes(encoding),"GBK"));


        });

3、newAPIHadoopFile+Configuration——亂碼

        JavaPairRDD<LongWritable, Text> longWritableTextJavaPairRDD = sc.newAPIHadoopFile(filePath, TextInputFormat.class, LongWritable.class, Text.class, entries );

        System.out.println("longWritableTextJavaPairRDD  count ="+longWritableTextJavaPairRDD.count());
        longWritableTextJavaPairRDD.foreach(k->{

            System.out.println(k._2);
        });

4、newAPIHadoopFile+自定義類——亂碼

        JavaPairRDD<LongWritable, Text> longWritableTextJavaPairRDD = sc.newAPIHadoopFile(filePath, GBKInputFormat.class, LongWritable.class, Text.class, entries );

        System.out.println("longWritableTextJavaPairRDD  count ="+longWritableTextJavaPairRDD.count());
        longWritableTextJavaPairRDD.foreach(k->{

            System.out.println(k._2);
        });

代碼中GBKInputFormat.class是TextInputFormat.class復制將內部UTF-8修改為GB18030所得

5、newAPIHadoopRDD+自定義類——亂碼

        JavaPairRDD<LongWritable, Text> longWritableTextJavaPairRDD1 = sc.newAPIHadoopRDD(entries, GBKInputFormat.class, LongWritable.class, Text.class);
        System.out.println("longWritableTextJavaPairRDD  count ="+longWritableTextJavaPairRDD1.count());
        longWritableTextJavaPairRDD1.foreach(k->{
            System.out.println(k._2());
        });

三、最終解決

上述方法感覺指定的字符編碼并沒有生效不知道為什么，如有了解原因的還請為我解惑，謝謝

最終解決方案如下

       JavaPairRDD<LongWritable, Text> longWritableTextJavaPairRDD = sc.newAPIHadoopFile(filePath, TextInputFormat.class, LongWritable.class, Text.class, new Configuration());

        System.out.println("longWritableTextJavaPairRDD  count ="+longWritableTextJavaPairRDD.count());
        longWritableTextJavaPairRDD.foreach(k->{
            System.out.println(new String(k._2.copyBytes(), encoding));
        });

        JavaPairRDD<LongWritable, Text> longWritableTextJavaPairRDD1 = sc.newAPIHadoopRDD(entries, TextInputFormat.class, LongWritable.class, Text.class);

        System.out.println("longWritableTextJavaPairRDD  count ="+longWritableTextJavaPairRDD1.count());
        longWritableTextJavaPairRDD1.foreach(k->{
            System.out.println(new String(k._2().copyBytes(),encoding));
            System.out.println(new String(k._2.copyBytes(),encoding));
        });

主要是new String(k._2().copyBytes(),encoding)得以解決

到此這篇關于java spark文件讀取亂碼問題的解決方法的文章就介紹到這了,更多相關java spark文件讀取亂碼內容請搜索腳本之家以前的文章或繼續(xù)瀏覽下面的相關文章希望大家以后多多支持腳本之家！

您可能感興趣的文章: