linux_HDFS節(jié)點(diǎn)缺失問題及解析

更新時間：2023年12月05日 09:26:17 作者：12級龍卷風(fēng)

這篇文章主要介紹了linux_HDFS節(jié)點(diǎn)缺失問題及解析,具有很好的參考價值,希望對大家有所幫助,如有錯誤或未考慮完全的地方,望不吝賜教

一、背景介紹

現(xiàn)在虛擬機(jī)node1，node2，node3

使用：start-dfs.sh 一鍵啟動HDFS集群

二、現(xiàn)象描述

1.查看node1的進(jìn)程啟動情況

[root@node1 ~]#start-dfs.sh
[root@node1 ~]# jps
4145 Jps
2102 NameNode
2247 DataNode
[root@node1 ~]#

2.查看node2的進(jìn)程啟動情況

[root@node2 logs]# jps
3828 Jps
1800 DataNode
[root@node2 logs]#

3.查看node3的進(jìn)程啟動情況

[root@node3 ~]# jps
3428 Jps
[root@node3 ~]#

發(fā)現(xiàn)問題：node3的datanode從節(jié)點(diǎn)沒有啟動

三、問題分析

1.一鍵啟動配置是否出現(xiàn)問題

一般系統(tǒng)配置不會出錯，第一次啟動成功就說明是好的。

2.異常：從節(jié)點(diǎn)自動關(guān)閉或掉線

2.1.通過查看node3日志去排查問題

日志路徑：/export/server/hadoop-3.3.0/logs/索要查看的日志文件

[root@node3 ~]# cd /export/server/hadoop-3.3.0/logs
[root@node3 logs]# cat hadoop-root-datanode-node3.itcast.cn.log
2023-06-06 07:24:31,896 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: RECEIVED SIGNAL 1: SIGHUP
2023-06-06 07:24:31,907 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: RECEIVED SIGNAL 15: SIGTERM
2023-06-06 07:24:31,914 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at node3.itcast.cn/192.168.88.153
************************************************************/
[root@node3 logs]#

注意：在查看日志的時候，上述方式會顯示從第一天啟動開始到現(xiàn)在的日志，這是一個很龐大的數(shù)據(jù)過程。

所以通過如下方式去查看node3的日志：

①vim 進(jìn)入到日志文件
②命令模式：通過/關(guān)鍵時間定位到固定日志查看

[root@node3 logs]# ll
total 7020
-rw-r--r-- 1 root root 6950235 Jun 6 11:31 hadoop-root-datanode-node3.itcast.cn.log
-rw-r--r-- 1 root root 692 Jun 6 11:26 hadoop-root-datanode-node3.itcast.cn.out
-rw-r--r-- 1 root root 692 Jun 5 21:33 hadoop-root-datanode-node3.itcast.cn.out.1
-rw-r--r-- 1 root root 692 Jun 5 18:15 hadoop-root-datanode-node3.itcast.cn.out.2
-rw-r--r-- 1 root root 692 Jun 5 07:15 hadoop-root-datanode-node3.itcast.cn.out.3
-rw-r--r-- 1 root root 692 Jun 4 22:44 hadoop-root-datanode-node3.itcast.cn.out.4
-rw-r--r-- 1 root root 205640 Jun 5 19:33 hadoop-root-nodemanager-node3.itcast.cn.log
-rw-r--r-- 1 root root 2201 Jun 5 18:19 hadoop-root-nodemanager-node3.itcast.cn.out
-rw-r--r-- 1 root root 2201 Jun 5 07:23 hadoop-root-nodemanager-node3.itcast.cn.out.1
-rw-r--r-- 1 root root 0 Jun 4 22:44 SecurityAuth-root.audit
drwxr-xr-x 2 root root 6 Jun 5 07:40 userlogs
[root@node3 logs]# vim hadoop-root-datanode-node3.itcast.cn.log
2023-06-05 21:33:37,872 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding block pool BP-389489230-192.168.88.151-1685888665811
2023-06-05 21:33:37,873 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Scanning block pool BP-389489230-192.168.88.151-1685888665811 on volume /export/data/hadoop-3.3.0/dfs/data...
2023-06-05 21:33:37,913 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time taken to scan block pool BP-389489230-192.168.88.151-1685888665811 on /export/data/hadoop-3.3.0/dfs/data: 41ms
2023-06-05 21:33:37,913 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Total time to scan all replicas for block pool BP-389489230-192.168.88.151-1685888665811: 42ms
2023-06-05 21:33:37,915 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding replicas to map for block pool BP-389489230-192.168.88.151-1685888665811 on volume /export/data/hadoop-3.3.0/dfs/data...
2023-06-05 21:33:37,915 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice: Replica Cache file: /export/data/hadoop-3.3.0/dfs/data/current/BP-389489230-192.168.88.151-1685888665811/current/replicas doesn't exist
2023-06-05 21:33:37,930 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time to add replicas to map for block pool BP-389489230-192.168.88.151-1685888665811 on volume /export/data/hadoop-3.3.0/dfs/data: 16ms
2023-06-05 21:33:37,930 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Total time to add all replicas to map for block pool BP-389489230-192.168.88.151-1685888665811: 16ms
2023-06-05 21:33:37,931 INFO org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker: Scheduling a check for /export/data/hadoop-3.3.0/dfs/data
2023-06-05 21:33:37,941 INFO org.apache.hadoop.hdfs.server.datanode.checker.DatasetVolumeChecker: Scheduled health check for volume /export/data/hadoop-3.3.0/dfs/data
2023-06-05 21:33:37,952 INFO org.apache.hadoop.hdfs.server.datanode.VolumeScanner: VolumeScanner(/export/data/hadoop-3.3.0/dfs/data, DS-151efa3b-8d41-483c-91d3-24c93d2871a2): no suitable block pools found to scan. Waiting 1732279321 ms.
2023-06-05 21:33:37,955 WARN org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value above 1000 ms/sec. Assuming default value of -1
2023-06-05 21:33:37,955 INFO org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: Periodic Directory Tree Verification scan starting in 7915082ms with interval of 21600000ms and throttle limit of -1ms/s
2023-06-05 21:33:37,961 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-389489230-192.168.88.151-1685888665811 (Datanode Uuid e7c67709-7680-4f33-81ed-50f6bbf48b46) service to node1/192.168.88.151:8020 beginning handshake with NN
2023-06-05 21:33:37,997 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-389489230-192.168.88.151-1685888665811 (Datanode Uuid e7c67709-7680-4f33-81ed-50f6bbf48b46) service to node1/192.168.88.151:8020 successfully registered with NN
2023-06-05 21:33:37,998 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: For namenode node1/192.168.88.151:8020 using BLOCKREPORT_INTERVAL of 21600000msecs CACHEREPORT_INTERVAL of 10000msecs Initial delay: 0msecs; heartBeatInterval=3000
2023-06-05 21:33:38,138 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Successfully sent block report 0x45e611d10c32cfb7, containing 1 storage report(s), of which we sent 1. The reports had 4 total blocks and used 1 RPC(s). This took 5 msecs to generate and 83 msecs for RPC and NN processing. Got back one command: FinalizeCommand/5.
2023-06-05 21:33:38,138 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Got finalize command for block pool BP-389489230-192.168.88.151-1685888665811
2023-06-06 07:24:31,896 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: RECEIVED SIGNAL 1: SIGHUP
2023-06-06 07:24:31,907 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: RECEIVED SIGNAL 15: SIGTERM
2023-06-06 07:24:31,914 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at node3.itcast.cn/192.168.88.153
************************************************************/
[root@node3 logs]# jps
3458 Jps
[root@node3 logs]# cd /export/server/hadoop-3.3.0/sbin
[root@node3 sbin]# ll
total 108
-rwxr-xr-x 1 root root 2756 Jun 4 21:10 distribute-exclude.sh
drwxr-xr-x 4 root root 36 Jun 4 21:10 FederationStateStore
-rwxr-xr-x 1 root root 1983 Jun 4 21:10 hadoop-daemon.sh
-rwxr-xr-x 1 root root 2522 Jun 4 21:10 hadoop-daemons.sh
-rwxr-xr-x 1 root root 1542 Jun 4 21:10 httpfs.sh
-rwxr-xr-x 1 root root 1500 Jun 4 21:10 kms.sh
-rwxr-xr-x 1 root root 1841 Jun 4 21:10 mr-jobhistory-daemon.sh
-rwxr-xr-x 1 root root 2086 Jun 4 21:10 refresh-namenodes.sh
-rwxr-xr-x 1 root root 1779 Jun 4 21:10 start-all.cmd
-rwxr-xr-x 1 root root 2221 Jun 4 21:10 start-all.sh
-rwxr-xr-x 1 root root 1880 Jun 4 21:10 start-balancer.sh
-rwxr-xr-x 1 root root 1401 Jun 4 21:10 start-dfs.cmd
-rwxr-xr-x 1 root root 5170 Jun 4 21:10 start-dfs.sh
-rwxr-xr-x 1 root root 1793 Jun 4 21:10 start-secure-dns.sh
-rwxr-xr-x 1 root root 1571 Jun 4 21:10 start-yarn.cmd
-rwxr-xr-x 1 root root 3342 Jun 4 21:10 start-yarn.sh
-rwxr-xr-x 1 root root 1770 Jun 4 21:10 stop-all.cmd
-rwxr-xr-x 1 root root 2166 Jun 4 21:10 stop-all.sh
-rwxr-xr-x 1 root root 1783 Jun 4 21:10 stop-balancer.sh
-rwxr-xr-x 1 root root 1455 Jun 4 21:10 stop-dfs.cmd
-rwxr-xr-x 1 root root 3898 Jun 4 21:10 stop-dfs.sh
-rwxr-xr-x 1 root root 1756 Jun 4 21:10 stop-secure-dns.sh
-rwxr-xr-x 1 root root 1642 Jun 4 21:10 stop-yarn.cmd
-rwxr-xr-x 1 root root 3083 Jun 4 21:10 stop-yarn.sh
-rwxr-xr-x 1 root root 1982 Jun 4 21:10 workers.sh
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:562)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1881)
at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1191)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1087)
2023-06-05 00:03:05,799 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: RECEIVED SIGNAL 15: SIGTERM
2023-06-05 00:03:05,805 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at node3.itcast.cn/192.168.88.153
************************************************************/
2023-06-05 07:15:43,274 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG: host = node3.itcast.cn/192.168.88.153
STARTUP_MSG: args = []
STARTUP_MSG: version = 3.3.0
@ @
/2023-06-05