Docker實現(xiàn)安裝ELK(單節(jié)點)

更新時間：2024年08月14日 16:37:56 作者：wang18057

這篇文章主要介紹了Docker實現(xiàn)安裝ELK(單節(jié)點),具有很好的參考價值,希望對大家有所幫助,如有錯誤或未考慮完全的地方,望不吝賜教

Docker 安裝ELK(單節(jié)點)

創(chuàng)建docker網(wǎng)絡(luò)

docker network create -d bridge elastic

拉取elasticsearch 8.4.3版本

docker pull docker.elastic.co/elasticsearch/elasticsearch:8.4.3
也可能是這個
docker pull elasticsearch:8.4.3

第一次執(zhí)行docker腳本

docker run -it \
-p 9200:9200 \
-p 9300:9300 \
--name elasticsearch \
--net elastic \
-e ES_JAVA_OPTS="-Xms1g -Xmx1g" \
-e "discovery.type=single-node" \
-e LANG=C.UTF-8 \
-e LC_ALL=C.UTF-8 \
elasticsearch:8.4.3

注意第一次執(zhí)行腳本不要加-d這個參數(shù)，否則看不到服務(wù)首次運行時生成的隨機密碼和隨機 enrollment token

拷貝日志中的以下內(nèi)容，備用

? Elasticsearch security features have been automatically configured!
? Authentication is enabled and cluster connections are encrypted.

??  Password for the elastic user (reset with `bin/elasticsearch-reset-password -u elastic`):
  =HjjCu=tj1orDTLJbWPv

??  HTTP CA certificate SHA-256 fingerprint:
  9204867e59a004b04c44a98d93c4609937ce3f14175a3eed7afa98ee31bbd4c2

??  Configure Kibana to use this cluster:
? Run Kibana and click the configuration link in the terminal when Kibana starts.
? Copy the following enrollment token and paste it into Kibana in your browser (valid for the next 30 minutes):
  eyJ2ZXIiOiI4LjQuMyIsImFkciI6WyIxNzIuMjIuMC4yOjkyMDAiXSwiZmdyIjoiOTIwNDg2N2U1OWEwMDRiMDRjNDRhOThkOTNjNDYwOTkzN2NlM2YxNDE3NWEzZWVkN2FmYTk4ZWUzMWJiZDRjMiIsImtleSI6Img0bGNvSkFCYkJnR1BQQXRtb3VZOnpCcjZQMUtZVFhHb1VDS2paazRHRHcifQ==

?? Configure other nodes to join this cluster:
? Copy the following enrollment token and start new Elasticsearch nodes with `bin/elasticsearch --enrollment-token <token>` (valid for the next 30 minutes):
  eyJ2ZXIiOiI4LjQuMyIsImFkciI6WyIxNzIuMjIuMC4yOjkyMDAiXSwiZmdyIjoiOTIwNDg2N2U1OWEwMDRiMDRjNDRhOThkOTNjNDYwOTkzN2NlM2YxNDE3NWEzZWVkN2FmYTk4ZWUzMWJiZDRjMiIsImtleSI6ImhZbGNvSkFCYkJnR1BQQXRtb3VLOjRZWlFkN1JIUk5PcVJqZTlsX2p6LXcifQ==

  If you're running in Docker, copy the enrollment token and run:
  `docker run -e "ENROLLMENT_TOKEN=<token>" docker.elastic.co/elasticsearch/elasticsearch:8.4.3`

創(chuàng)建相應(yīng)目錄并復(fù)制配置文件到主機

mkdir -p apps/elk8.4.3/elasticsearch
# 這個cp命令是在 /home/ubuntu目錄下執(zhí)行的
docker cp elasticsearch:/usr/share/elasticsearch/config apps/elk8.4.3/elasticsearch/

docker cp elasticsearch:/usr/share/elasticsearch/data apps/elk8.4.3/elasticsearch/

docker cp elasticsearch:/usr/share/elasticsearch/plugins apps/elk8.4.3/elasticsearch/

docker cp elasticsearch:/usr/share/elasticsearch/logs apps/elk8.4.3/elasticsearch/

刪除容器

docker rm -f elasticsearch

修改apps/elk8.4.3/elasticsearch/config/elasticsearch.yml

vim apps/elk8.4.3/elasticsearch/config/elasticsearch.yml

添加

增加：xpack.monitoring.collection.enabled: true
說明：添加這個配置以后在kibana中才會顯示聯(lián)機狀態(tài)，否則會顯示脫機狀態(tài)

啟動elasticsearch

docker run -it \
-d \
-p 9200:9200 \
-p 9300:9300 \
--name elasticsearch \
--net elastic \
-e ES_JAVA_OPTS="-Xms1g -Xmx1g" \
-e "discovery.type=single-node" \
-e LANG=C.UTF-8 \
-e LC_ALL=C.UTF-8 \
-v /home/ubuntu/apps/elk8.4.3/elasticsearch/config:/usr/share/elasticsearch/config \
-v /home/ubuntu/apps/elk8.4.3/elasticsearch/data:/usr/share/elasticsearch/data \
-v /home/ubuntu/apps/elk8.4.3/elasticsearch/plugins:/usr/share/elasticsearch/plugins \
-v /home/ubuntu/apps/elk8.4.3/elasticsearch/logs:/usr/share/elasticsearch/logs \
elasticsearch:8.4.3

啟動驗證

https://xxxxx:9200/
用戶名:elastic
密碼在第一次啟動時保存下來的信息中查找

Kibana

安裝Kibana

docker pull kibana:8.4.3

啟動Kibana

docker run -it \
--restart=always \
--log-driver json-file \
--log-opt max-size=100m \
--log-opt max-file=2 \
--name kibana \
-p 5601:5601 \
--net elastic \
kibana:8.4.3

初始化Kibana鑒權(quán)憑證

http://xxxx:5601/?code=878708

注意：

在textarea中填入之前elasticsearch生成的相關(guān)信息,注意這個token只有30分鐘的有效期，如果過期了只能進入容器重置token,進入容器執(zhí)行

/bin/elasticsearch-create-enrollment-token -s kibana --url "https://127.0.0.1:9200"

kibana驗證

將服務(wù)端的log中輸出的驗證碼，輸入到瀏覽器中，我這里是628503

創(chuàng)建kibana目錄并copy相關(guān)配置信息

mkdir apps/elk8.4.3/kibana

# 這個cp命令是在 /home/ubuntu目錄下執(zhí)行的
docker cp kibana:/usr/share/kibana/config apps/elk8.4.3/kibana/


docker cp kibana:/usr/share/kibana/data apps/elk8.4.3/kibana/

docker cp kibana:/usr/share/kibana/plugins apps/elk8.4.3/kibana/

docker cp kibana:/usr/share/kibana/logs apps/elk8.4.3/kibana/

sudo chown -R 1000:1000 apps/elk8.4.3/kibana

修改apps/elk8.4.3/kibana/config/kibana.yml

### >>>>>>> BACKUP START: Kibana interactive setup (2024-03-25T07:30:11.689Z)

#
# ** THIS IS AN AUTO-GENERATED FILE **
#

# Default Kibana configuration for docker target
#server.host: "0.0.0.0"
#server.shutdownTimeout: "5s"
#elasticsearch.hosts: [ "http://elasticsearch:9200" ]
#monitoring.ui.container.elasticsearch.enabled: true
### >>>>>>> BACKUP END: Kibana interactive setup (2024-03-25T07:30:11.689Z)

# This section was automatically generated during setup.
i18n.locale: "zh-CN"
server.host: 0.0.0.0
server.shutdownTimeout: 5s
# #這個ip一定是elasticsearch的容器ip，可使用docker inspect | grep -i ipaddress
elasticsearch.hosts: ['https://your ip:9200']
monitoring.ui.container.elasticsearch.enabled: true
elasticsearch.serviceAccountToken: AAEAAWVsYXN0aWMva2liYW5hL2Vucm9sbC1wcm9jZXNzLXRva2VuLTE3MTEzNTE4MTA5NDM6ZHZ1R3M5cV9RRlc2NmQ3dE9WaWM0QQ
elasticsearch.ssl.certificateAuthorities: [/usr/share/kibana/data/ca_1711351811685.crt]
xpack.fleet.outputs: [{id: fleet-default-output, name: default, is_default: true, is_default_monitoring: true, type: elasticsearch, hosts: ['https://your ip:9200'], ca_trusted_fingerprint: 5e7d9fe48c485c2761f9e7a99b9d5737e4e34dc55b9bf6929d929fb34d61a11a}]

刪除容器并重啟

docker rm -f kibana

docker run -it \
-d \
--restart=always \
--log-driver json-file \
--log-opt max-size=100m \
--log-opt max-file=2 \
--name kibana \
-p 5601:5601 \
--net elastic \
-v /home/ubuntu/apps/elk8.4.3/kibana/config:/usr/share/kibana/config \
-v /home/ubuntu/apps/elk8.4.3/kibana/data:/usr/share/kibana/data \
-v /home/ubuntu/apps/elk8.4.3/kibana/plugins:/usr/share/kibana/plugins \
-v /home/ubuntu/apps/elk8.4.3/kibana/logs:/usr/share/kibana/logs \
kibana:8.4.3

Logstash

Logstash拉取鏡像

docker pull logstash:8.4.3

啟動

docker run -it \
-d \
--name logstash \
-p 9600:9600 \
-p 5044:5044 \
--net elastic \
logstash:8.4.3

創(chuàng)建目錄并同步配置文件

mkdir apps/elk8.4.3/logstash

# 這個cp命令是在 /home/ubuntu目錄下執(zhí)行的
docker cp logstash:/usr/share/logstash/config apps/elk8.4.3/logstash/ 
docker cp logstash:/usr/share/logstash/pipeline apps/elk8.4.3/logstash/ 

sudo cp -rf apps/elk8.4.3/elasticsearch/config/certs apps/elk8.4.3/logstash/config/certs

sudo chown -R 1000:1000 apps/elk8.4.3/logstash

修改配置apps/elk8.4.3/logstash/config/logstash.yml

http.host: "0.0.0.0"
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.hosts: [ "http://your ip:9200" ]
xpack.monitoring.elasticsearch.username: "elastic"
# 第一次啟動elasticsearch是保存的信息中查找 L3WKr6ROTiK_DbqzBr8c
xpack.monitoring.elasticsearch.password: "L3WKr6ROTiK_DbqzBr8c"
xpack.monitoring.elasticsearch.ssl.certificate_authority: "/usr/share/logstash/config/certs/http_ca.crt"
# 第一次啟動elasticsearch是保存的信息中查找 5e7d9fe48c485c2761f9e7a99b9d5737e4e34dc55b9bf6929d929fb34d61a11a
xpack.monitoring.elasticsearch.ssl.ca_trusted_fingerprint: "5e7d9fe48c485c2761f9e7a99b9d5737e4e34dc55b9bf6929d929fb34d61a11a"

修改配置apps/elk8.4.3/logstash/pipeline/logstash.conf

input {
  beats {
    port => 5044
  }
}


filter {
  date {
        # 因為我的日志里，我的time字段格式是2024-03-14T15:34:03+08:00 ，所以要使用以下兩行配置
        match => [ "time", "ISO8601" ]
        target => "@timestamp"
  }
  json {
    source => "message"
  }
  mutate {
    remove_field => ["message", "path", "version", "@version", "agent", "cloud", "host", "input", "log", "tags", "_index", "_source", "ecs", "event"]
  }
}


output {
  elasticsearch {
    hosts => ["https://your ip:9200"]
    index => "douyin-%{+YYYY.MM.dd}"
    ssl => true
    ssl_certificate_verification => false
    cacert => "/usr/share/logstash/config/certs/http_ca.crt"
    ca_trusted_fingerprint => "第一次啟動elasticsearch是保存的信息中查找e924551c1453c893114a05656882eea81cb11dd87c1258f83e6f676d2428f8f2"
    user => "elastic"
    password => "第一次啟動elasticsearch是保存的信息中查找UkNx8px1yrMYIht30QUc"
  }
}

刪除容器并重新啟動

docker rm -f logstash

docker run -it \
-d \
--name logstash \
-p 9600:9600 \
-p 5044:5044 \
--net elastic \
-v /home/ubuntu/apps/elk8.4.3/logstash/config:/usr/share/logstash/config \
-v /home/ubuntu/apps/elk8.4.3/logstash/pipeline:/usr/share/logstash/pipeline \
logstash:8.4.3

Filebeat

Filebeat拉取鏡像

sudo docker pull elastic/filebeat:8.4.3

FileBeat啟動

docker run -it \
-d \
--name filebeat \
--network elastic \
-e TZ=Asia/Shanghai \
elastic/filebeat:8.4.3 \
filebeat -e  -c /usr/share/filebeat/filebeat.yml


docker run -d --name filebeat \
  -v /home/linyanbo/docker_data/filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml \
  -v /home/linyanbo/docker_data/filebeat/data:/usr/share/filebeat/data \
  -v /var/logs/:/var/log \
  --link elasticsearch:elasticsearch \
  --network elastic \
  --user root \
 elastic/filebeat:8.4.3

設(shè)置開機啟動

docker update elasticsearch --restart=always

配置文件

filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/logs/duty-admin/spring.log/crmduty-admin-2024-07-12.log
  fields:
    log_source: oh-promotion
  fields_under_root: true
  multiline.pattern: ^\d{4}-\d{1,2}-\d{1,2}
  multiline.negate: true
  multiline.match: after
  scan_frequency: 5s
  close_inactive: 1h
  ignore_older: 24h

output.logstash:
  hosts: ["your ip:5044"]

logstash.conf

input {
  beats {
    port => 5044
  }
}
filter {
#	mutate {
#		split => {"message"=>" "}
#	}
	mutate {
		add_field => {
			"mm" => "%{message}"
		}
	}

}

output {
  elasticsearch {
    hosts => ["https://your ip:9200"]
    #index => "duty-admin%{+YYYY.MM.dd}"
    index => "duty-admin%{+YYYY}"
    ssl => true
    ssl_certificate_verification => false
    cacert => "/usr/share/logstash/config/certs/http_ca.crt"
    ca_trusted_fingerprint => "9204867e59a004b04c44a98d93c4609937ce3f14175a3eed7afa98ee31bbd4c2"
    user => "elastic"
    password => "=HjjCu=tj1orDTLJbWPv"
  }
}

 output {
  stdout {
    codec => rubydebug
  }
 }

elasticsearch.yml

cluster.name: "docker-cluster"
network.host: 0.0.0.0

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically      
# generated to configure Elasticsearch security features on 11-07-2024 05:54:41
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

# 說明：添加這個配置以后在kibana中才會顯示聯(lián)機狀態(tài)，否則會顯示脫機狀態(tài)
xpack.monitoring.collection.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

kibana.yml

### >>>>>>> BACKUP START: Kibana interactive setup (2024-07-11T06:09:35.897Z)

#
# ** THIS IS AN AUTO-GENERATED FILE **
#

# Default Kibana configuration for docker target
#server.host: "0.0.0.0"
#server.shutdownTimeout: "5s"
#elasticsearch.hosts: [ "http://elasticsearch:9200" ]
#monitoring.ui.container.elasticsearch.enabled: true
### >>>>>>> BACKUP END: Kibana interactive setup (2024-07-11T06:09:35.897Z)

# This section was automatically generated during setup.
server.host: 0.0.0.0
server.shutdownTimeout: 5s
elasticsearch.hosts: ['https://your ip:9200']
monitoring.ui.container.elasticsearch.enabled: true
elasticsearch.serviceAccountToken: AAEAAWVsYXN0aWMva2liYW5hL2Vucm9sbC1wcm9jZXNzLXRva2VuLTE3MjA2NzgxNzU2MzU6bU5RR25uQUVSaWExbUdHQ2tsODRmZw
elasticsearch.ssl.certificateAuthorities: [/usr/share/kibana/data/ca_1720678175894.crt]
xpack.fleet.outputs: [{id: fleet-default-output, name: default, is_default: true, is_default_monitoring: true, type: elasticsearch, hosts: ['https://your ip:9200'], ca_trusted_fingerprint: 9204867e59a004b04c44a98d93c4609937ce3f14175a3eed7afa98ee31bbd4c2}]