K8S使用NFS動態(tài)創(chuàng)建PVC實踐
更新時間：2026年03月03日 09:51:44 作者：一直學下去
本文介紹了兩種使用NFS作為K8S中容器數(shù)據(jù)持久化后端存儲的方法,并在產(chǎn)線環(huán)境中進行了驗證,第一種方法使用了nfs-client-provisioner,第二種方法使用了csi-nfs-driver,兩種方法均通過動態(tài)創(chuàng)建PVC實現(xiàn)數(shù)據(jù)持久化,并驗證了文件的讀寫能力
簡介
K8S中容器的數(shù)據(jù)的持久化對于有狀態(tài)的應(yīng)用是很重要的，考慮到橫向擴展，所以一般都會使用網(wǎng)絡(luò)存儲來作為持久化的介質(zhì)。NFS（有的叫NAS）就是經(jīng)常使用的一種。
本文介紹兩種使用NFS作為后端存儲，動態(tài)創(chuàng)建PVC的方法。兩種方法我在產(chǎn)線環(huán)境都驗證過。
- 第一種: nfs-client-provisioner (較老)
- 第二種: csi-nfs-driver （較新）
準備
Kubernetes : 1.17.6
實施
nfs-client-provisioner
1. 部署nfs-client-provisioner , 文件里面的nfs的服務(wù)器地址和路徑換成自己的
kubectl apply -f nfs-storageclass-allInOne.yaml
kind: ServiceAccount
apiVersion: v1
metadata:
name: nfs-client-provisioner
namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nfs-client-provisioner-runner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: run-nfs-client-provisioner
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
namespace: kube-system
roleRef:
kind: ClusterRole
name: nfs-client-provisioner-runner
apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: kube-system
name: leader-locking-nfs-client-provisioner
rules:
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: kube-system
name: leader-locking-nfs-client-provisioner
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: kube-system
roleRef:
kind: Role
name: leader-locking-nfs-client-provisioner
apiGroup: rbac.authorization.k8s.io
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: nfs-client-provisioner
namespace: kube-system
spec:
replicas: 2
strategy:
type: Recreate
selector:
matchLabels:
app: nfs-client-provisioner
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccountName: nfs-client-provisioner
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
topologyKey: kubernetes.io/hostname
labelSelector:
matchLabels:
app: nfs-client-provisioner
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: dedicated
# operator: In
# values:
# - "cmp"
containers:
- name: nfs-client-provisioner
image: quay.io/external_storage/nfs-client-provisioner:v3.1.0-k8s1.11
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: nfs-client-provisioner
- name: NFS_SERVER
value: 10.159.1.105 #nfs服務(wù)器的地址
- name: NFS_PATH
value: /data/nfs_basedir/ #nfs的export 的目錄
volumes:
- name: nfs-client-root
nfs:
server: 10.159.1.105 #nfs服務(wù)器的地址
path: /data/nfs_basedir/ #nfs的export 的目錄
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs
annotations:
storageclass.kubernetes.io/is-default-class: true #是否是默認的SC
provisioner: nfs-client-provisioner
reclaimPolicy: Delete #有兩種選擇, Delete是PV刪除以后,文件也從NFS里面刪除；Retain是保留文件
mountOptions:
- noresvport
- vers=3
parameters:
archiveOnDelete: "false" #刪除的時候是否歸檔，如果是true就是軟刪除，只是會重命名pv的文件目錄,這樣會造成很多垃圾數(shù)據(jù)
2. 使用新創(chuàng)建的StorageClass，動態(tài)創(chuàng)建PVC
kubectl apply -f test-nfs-sc.yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
containers:
- name: nginx
image: nginx:latest
ports:
- containerPort: 80
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
volumes:
- name: www
persistentVolumeClaim:
claimName: nginx
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nginx
spec:
storageClassName: "nfs"
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi3. 驗證PVC是否創(chuàng)建成功。進入到容器中，創(chuàng)建文件，然后把nfs掛載到本地，看看文件是否存在
kubectl get pvc
[root@cmp-k8s-prd-10-128-148-230 nfs-ksyun]# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE nginx Bound pvc-b58182db-a9ca-434b-88ae-6d28c9795242 5Gi RWX nfs 2s
csi-nfs-driver
1. 部署csi-nfs-driver
kubectl apply -f rbac-csi-nfs-controller.yaml
rbac-csi-nfs-controller.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: csi-nfs-controller-sa
namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nfs-external-provisioner-role
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
- apiGroups: ["storage.k8s.io"]
resources: ["csinodes"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "watch"]
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nfs-csi-provisioner-binding
subjects:
- kind: ServiceAccount
name: csi-nfs-controller-sa
namespace: kube-system
roleRef:
kind: ClusterRole
name: nfs-external-provisioner-role
apiGroup: rbac.authorization.k8s.iokubectl apply -f csi-nfs-driverinfo.yaml kubectl apply -f csi-nfs-node.yaml kubectl apply -f csi-nfs-controller.yaml kubectl apply -f storageclass-nfs.yaml
csi-nfs-driverinfo.yaml
---
apiVersion: storage.k8s.io/v1beta1
kind: CSIDriver
metadata:
name: nfs.csi.k8s.io
spec:
attachRequired: false
volumeLifecycleModes:
- Persistent
podInfoOnMount: truecsi-nfs-node.yaml
---
# This YAML file contains driver-registrar & csi driver nodeplugin API objects
# that are necessary to run CSI nodeplugin for nfs
kind: DaemonSet
apiVersion: apps/v1
metadata:
name: csi-nfs-node
namespace: kube-system
spec:
selector:
matchLabels:
app: csi-nfs-node
template:
metadata:
labels:
app: csi-nfs-node
spec:
hostNetwork: true # original nfs connection would be broken without hostNetwork setting
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: liveness-probe
image: registry.cn-hangzhou.aliyuncs.com/imagesfromgoogle/livenessprobe:v2.1.0
args:
- --csi-address=/csi/csi.sock
- --probe-timeout=3s
- --health-port=29653
- --v=5
volumeMounts:
- name: socket-dir
mountPath: /csi
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 20Mi
- name: node-driver-registrar
image: registry.cn-hangzhou.aliyuncs.com/imagesfromgoogle/csi-node-driver-registrar:v2.0.1
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "rm -rf /registration/csi-nfsplugin /registration/csi-nfsplugin-reg.sock"]
args:
- --v=5
- --csi-address=/csi/csi.sock
- --kubelet-registration-path=/var/lib/kubelet/plugins/csi-nfsplugin/csi.sock
env:
- name: KUBE_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
volumeMounts:
- name: socket-dir
mountPath: /csi
- name: registration-dir
mountPath: /registration
- name: nfs
securityContext:
privileged: true
capabilities:
add: ["SYS_ADMIN"]
allowPrivilegeEscalation: true
image: registry.cn-hangzhou.aliyuncs.com/imagesfromgoogle/nfsplugin:amd64-linux-canary
args:
- "-v=5"
- "--nodeid=$(NODE_ID)"
- "--endpoint=$(CSI_ENDPOINT)"
env:
- name: NODE_ID
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: CSI_ENDPOINT
value: unix:///csi/csi.sock
imagePullPolicy: "IfNotPresent"
volumeMounts:
- name: socket-dir
mountPath: /csi
- name: pods-mount-dir
mountPath: /var/lib/kubelet/pods
mountPropagation: "Bidirectional"
volumes:
- name: socket-dir
hostPath:
path: /var/lib/kubelet/plugins/csi-nfsplugin
type: DirectoryOrCreate
- name: pods-mount-dir
hostPath:
path: /var/lib/kubelet/pods
type: Directory
- hostPath:
path: /var/lib/kubelet/plugins_registry
type: Directory
name: registration-dircsi-nfs-controller.yaml
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: csi-nfs-controller
namespace: kube-system
spec:
replicas: 2
selector:
matchLabels:
app: csi-nfs-controller
template:
metadata:
labels:
app: csi-nfs-controller
spec:
serviceAccountName: csi-nfs-controller-sa
nodeSelector:
kubernetes.io/os: linux
priorityClassName: system-cluster-critical
tolerations:
- key: "node-role.kubernetes.io/master"
operator: "Equal"
value: "true"
effect: "NoSchedule"
containers:
- name: csi-provisioner
image: registry.cn-hangzhou.aliyuncs.com/imagesfromgoogle/csi-provisioner:v2.0.4
args:
- "-v=5"
- "--csi-address=$(ADDRESS)"
- "--leader-election"
env:
- name: ADDRESS
value: /csi/csi.sock
volumeMounts:
- mountPath: /csi
name: socket-dir
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 20Mi
- name: liveness-probe
image: registry.cn-hangzhou.aliyuncs.com/imagesfromgoogle/livenessprobe:v2.1.0
args:
- --csi-address=/csi/csi.sock
- --probe-timeout=3s
- --health-port=29652
- --v=5
volumeMounts:
- name: socket-dir
mountPath: /csi
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 20Mi
- name: nfs
image: registry.cn-hangzhou.aliyuncs.com/imagesfromgoogle/nfsplugin:amd64-linux-canary
securityContext:
privileged: true
capabilities:
add: ["SYS_ADMIN"]
allowPrivilegeEscalation: true
imagePullPolicy: IfNotPresent
args:
- "-v=5"
- "--nodeid=$(NODE_ID)"
- "--endpoint=$(CSI_ENDPOINT)"
env:
- name: NODE_ID
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: CSI_ENDPOINT
value: unix:///csi/csi.sock
volumeMounts:
- name: pods-mount-dir
mountPath: /var/lib/kubelet/pods
mountPropagation: "Bidirectional"
- mountPath: /csi
name: socket-dir
resources:
limits:
cpu: 200m
memory: 200Mi
requests:
cpu: 10m
memory: 20Mi
volumes:
- name: pods-mount-dir
hostPath:
path: /var/lib/kubelet/pods
type: Directory
- name: socket-dir
emptyDir: {}storageclass-nfs.yaml ,修改以下的nfs的地址和目錄
--- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: nfs-csi provisioner: nfs.csi.k8s.io parameters: server: 10.159.1.105 #修改為自己的nfs的服務(wù)器地址 share: /data/nfs_basedir/ #修改為nfs的目錄 reclaimPolicy: Retain # only retain is supported,目前這個回收策略只支持Retain volumeBindingMode: Immediate mountOptions: - vers=3 - noresvport
2. 查看部署的pod是否都啟動了
kubectl get po -n kube-system|grep csi-nfs
root@workstation:/data/nfs-csi# kubectl get po -n kube-system|grep csi-nfs csi-nfs-controller-59697d8b6-lm5v8 3/3 Running 0 5h35m csi-nfs-controller-59697d8b6-pbv5k 3/3 Running 0 5h35m csi-nfs-node-5zvfh 3/3 Running 0 5h35m csi-nfs-node-8qcw6 3/3 Running 0 5h35m csi-nfs-node-sjf8m 3/3 Running 0 5h35m csi-nfs-node-tkq4w 3/3 Running 0 5h35m csi-nfs-node-xls9l 3/3 Running 3 5h35m
3. 驗證新的SC已經(jīng)創(chuàng)建成功
kubectl get sc root@workstation:/data/rke/mercury_k8s/nfs-storage-class/nfs-csi# kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE nfs-csi nfs.csi.k8s.io Retain Immediate false 154m
4. 使用創(chuàng)建的SC,動態(tài)創(chuàng)建PVC
kubectl apply -f test-csi-nfs-driver.yaml
---
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
app: nginx
spec:
ports:
- port: 80
name: web
clusterIP: None
selector:
app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: web
spec:
selector:
matchLabels:
app: nginx
serviceName: "nginx"
replicas: 1
template:
metadata:
labels:
app: nginx
spec:
terminationGracePeriodSeconds: 10
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: www
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "nfs-csi"
resources:
requests:
storage: 1Gi5.查看新創(chuàng)建的PVC，可以登錄容器創(chuàng)建文件，然后把nfs掛在本地查看文件。這個操作就不演示了。
root@workstation:/data/nfs-csi# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE www-web-0 Bound pvc-9c6e135e-8df4-4d50-aad2-ed8e054f5c12 1Gi RWO nfs-csi 2m52s
總結(jié)
以上為個人經(jīng)驗，希望能給大家一個參考，也希望大家多多支持腳本之家。
相關(guān)文章
Kubernetes中Volume用于在Pod中提供共享存儲,生命周期與Pod解耦,支持多種類型如emptyDir、hostPath、NFS和PersistentVolumeClaim,每種Volume類型適用于不同的場景,如臨時存儲、宿主機文件訪問、共享數(shù)據(jù)和持久化存儲,通過示例展示了如何創(chuàng)建和使用這些Volume2025-12-12
本文主要介紹了阿里云oss對象存儲使用詳細步驟，文中通過示例代碼介紹的非常詳細，對大家的學習或者工作具有一定的參考學習價值，需要的朋友們下面隨著小編來一起學習學習吧2022-06-06
K8S?實用工具之合并多個kubeconfig實現(xiàn)詳解
這篇文章主要為大家介紹了K8S?實用工具之合并多個kubeconfig實現(xiàn)詳解，有需要的朋友可以借鑒參考下，希望能夠有所幫助，祝大家多多進步，早日升職加薪2023-03-03
本文主要介紹了Kubernetes中ConfigMap和Secret的使用方法,以及如何在Pod和容器中進行資源配置,文中詳細講解了如何創(chuàng)建和使用ConfigMap來管理非機密性配置,以及如何使用Secret來存儲敏感信息,同時,還介紹了如何在Pod中配置資源請求和限制,感興趣的朋友一起看看吧2025-03-03
Kubernetes資源通過GKV（Group、Version、Kind）標識,用于分類、版本管理和資源識別,在YAML配置中,apiVersion和kind字段分別體現(xiàn)GKV,是K8s資源操作的基礎(chǔ),通過GKV,Kubernetes可以正確解析和執(zhí)行資源配置,并確保版本兼容性2025-11-11
在Kubernetes中配置Ingress的SSL/TLS證書涉及兩個主要步驟：首先是創(chuàng)建包含證書的Secret,然后是配置Ingress資源以使用該Secret,這要求已部署IngressController如NGINX,并擁有有效的SSL/TLS證書,可以通過Cert-Manager自動管理證書的申請和續(xù)期2025-10-10
這篇文章主要為大家介紹了基于k8s部署Session模式Flink集群詳解，有需要的朋友可以借鑒參考下，希望能夠有所幫助，祝大家多多進步，早日升職加薪2023-03-03
這篇文章主要介紹了kubernetes?k8s?CRD自定義資源學習筆記，有需要的朋友可以借鑒參考下，希望能夠有所幫助，祝大家多多進步，早日升職加薪2022-05-05
K8S如何利用Prometheus監(jiān)控pod的實時數(shù)據(jù)指標
這篇文章主要給大家介紹了關(guān)于K8S如何利用Prometheus監(jiān)控pod的實時數(shù)據(jù)指標的相關(guān)資料,Prometheus是一個開源的服務(wù)監(jiān)控系統(tǒng)和時序數(shù)據(jù)庫,其提供了通用的數(shù)據(jù)模型和快捷數(shù)據(jù)采集、存儲和查詢接口,需要的朋友可以參考下2024-01-01
這篇文章主要介紹了K8S中設(shè)置JVM堆棧大小實現(xiàn)方式,具有很好的參考價值,希望對大家有所幫助,如有錯誤或未考慮完全的地方,望不吝賜教2025-07-07