1. 什么是 Velero?
Velero 是一个 Vmware 大厂用 Go 语言开发的开源工具，用于备份和恢复 Kubernetes 集群中的资源和持久化数据。
它支持在公有云平台或本地运行，提供备份集群数据、集群资源迁移到其他集群、以及将生产集群复制到开发和测试集群的能力。
2. Velero 与 ETCD 快照备份的区别
-
备份粒度：ETCD 快照是全局备份，而 Velero 可以对 Kubernetes 集群内对象级别进行备份，支持按 Type、Namespace、Label 等对象进行分类备份或恢复。
-
恢复影响：ETCD 快照恢复会影响其他 namespace 中的 pod 运行服务，而 Velero 可以针对性地恢复，不影响其他服务。
-
存储位置：Velero 支持多种对象存储，如 MiniO、Ceph、OSS、S3等存储，而 ETCD 快照是一个本地文件。
-
备份方式：Velero 支持增量备份，只备份变化的数据，而 ETCD 快照通常是全量备份。
3. Velero 整体架构
Velero 由客户端和服务端两部分组成。客户端是本地运行的命令行工具，服务端运行在 Kubernetes 集群中，负责执行具体的备份和恢复操作。
4. Velero 备份工作流程
-
本地 Velero 客户端发送备份指令。
-
Kubernetes 集群内创建一个 Backup 对象。
-
BackupController 监测 Backup 对象并开始备份过程。
-
BackupController 向 API Server 查询相关数据。
-
BackupController 将查询到的数据备份到远端的对象存储（MiniO）。
5. 对象存储MiniO
当前 K8S 集群中，已经有完整 MiniO 服务，部署步骤略；
root@k8s-master:/# kubectl get pod -n minio-system
NAME READY STATUS RESTARTS AGE
minio-0 1/1 Running 0 41h
minio-1 1/1 Running 0 41h
在 MiniO 集群中创建 S3桶
mc mb minio/velero-k8s
6. 部署Velero
安装 Velero 客户端（最新版 1.15）
wget https://github.com/vmware-tanzu/velero/releases/download/v1.14.0/velero-v1.14.0-linux-amd64.tar.gz
tar -zxvf velero-v1.14.0-linux-amd64.tar.gz
cp velero-v1.14.0-linux-amd64/velero /usr/local/bin/
velero version
创建 Velero 使用的 MiniO 密钥
[root@master1 /root]# cat >>credentials-velero <EOF
[default]
aws_access_key_id = admin
aws_secret_access_key = XXXXXXX
EOF
创建 Velero namespace
kubectl create ns velero
在K8S中安装 Velero 服务端
velero install \# 指定存储提供商，minio用的是s3，所以这里是aws--provider aws \# 指定aws插件--plugins velero/velero-plugin-for-aws:v1.10.1 \# 指定velero镜像--image velero/velero:v1.14.1 \# 限制agent CPU和内存资源--node-agent-pod-cpu-limit 2 \--node-agent-pod-mem-limit 2048Mi \# 指定命名空间--namespace velero \# 指定bucket名称--bucket velero-k8s \# 默认以文件形式备份持久化卷--default-volumes-to-fs-backup \# 启用node agent 进行备份和恢复操作，--use-restic已启用，从v1.10开始默认使用kopia备份--use-node-agent \# 指定S3存储桶凭证--secret-file ./credentials-velero \# 禁用卷快照功能，只使用文件系统级别的备份--use-volume-snapshots=false \# 配置备份存储位置的详细信息，其中<minio-api-url>填写minio的api地址，如192.168.10.25:9000--backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://<minio-api-url>
7. 无状态备份
#【1】在demo/demoapp 创建app
[root@master1 /root]# kubectl create ns demo
namespace/demo created
[root@master1 /root]# kubectl create deployment demoapp --image=ikubernetes/demoapp:v1.0 --replicas=3 -n demo
deployment.apps/demoapp created
[root@master1 /root]# kubectl get pods -n demo
NAME READY STATUS RESTARTS AGE
demoapp-7c58cd6bb-cs2fc 1/1 Running 0 49s
demoapp-7c58cd6bb-r4rcb 1/1 Running 0 49s
demoapp-7c58cd6bb-w7ckj 1/1 Running 0 49s#【2】创建一个备份 - 只备份 demo名称空间
[root@master1 /root]# velero backup create demo --include-namespaces demo
Backup request "demo" submitted successfully.
Run `velero backup describe demo` or `velero backup logs demo` for more details.[root@master1 /root]# velero backup get
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR
demo Completed 0 0 2024-11-21 06:46:30 +0000 UTC 29d default <none>#【3】模拟误删除demo名称空间 进行恢复操作
[root@master1 /root]# kubectl delete ns demo
[root@master1 /root]# kubectl get ns #确定无demo名称空间
NAME STATUS AGE
cilium-secrets Active 20d
default Active 21d
kube-node-lease Active 21d
kube-public Active 21d
kube-system Active 21d
metallb-system Active 20d
velero Active 40m#进行恢复
[root@master1 /root]# velero restore create --from-backup demo
Restore request "demo-20240321065255" submitted successfully.
Run `velero restore describe demo-20240321065255` or `velero restore logs demo-20240321065255` for more details.[root@master1 /root]# velero restore get
NAME BACKUP STATUS STARTED COMPLETED ERRORS WARNINGS CREATED SELECTOR
demo-20240321065255 demo Completed 2024-11-21 06:52:55 +0000 UTC 2024-11-21 06:52:55 +0000 UTC 0 2 2024-11-21 06:52:55 +0000 UTC <none>
[root@master1 /root]# kubectl get ns #已恢复demo名称空间
NAME STATUS AGE
cilium-secrets Active 20d
default Active 21d
demo Active 23s
kube-node-lease Active 21d
kube-public Active 21d
kube-system Active 21d
metallb-system Active 20d
velero Active 42m[root@master1 /root]# kubectl get pods -n demo #pod重新拉起
NAME READY STATUS RESTARTS AGE
demoapp-7c58cd6bb-cs2fc 1/1 Running 0 36s
demoapp-7c58cd6bb-r4rcb 1/1 Running 0 36s
demoapp-7c58cd6bb-w7ckj 1/1 Running 0 36s
8. 有状态备份
背景：创建一个有状态redis应用，底层数据持久化使用NFS，验证数据的备份恢复
#【1】验证集群storageclass
[root@master1 /root]# kg sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
nfs-csi (default) fuseim.pri/ifs Delete Immediate false 466d#【2】创建demo/redis 使用nfs-pvc持久化
[root@master1 /root]# cat redis-with-nfs-pvc.yaml
apiVersion: v1
kind: Pod
metadata:name: redis-with-nfs-pvc
spec:containers:- name: redisimage: redis:7-alpineports:- containerPort: 6379name: redisvolumeMounts:- mountPath: /dataname: data-storagevolumes:- name: data-storagepersistentVolumeClaim:claimName: nfs-csi[root@master1 /root]# kubectl apply -f redis-with-nfs-pvc.yaml -n demo
pod/redis-with-nfs-pvc created[root@master1 /root]# kubectl get pods -n demo
NAME READY STATUS RESTARTS AGE
redis-with-nfs-pvc 1/1 Running 0 51s#【3】测试写数据
[root@master1 /root]# kubectl exec -it redis-with-nfs-pvc -n demo -- /bin/sh
/data # redis-cli
127.0.0.1:6379> set mykey "BIRKHOFF 2024-11-21"
OK
127.0.0.1:6379> BGSAVE
Background saving started
127.0.0.1:6379> exit
/data # ls
dump.rdb#【4】备份数据 - redis-backup
[root@master1 /root]# velero backup create redis-backup --include-namespaces demo --default-volumes-to-fs-backup
Backup request "redis-backup" submitted successfully.
Run `velero backup describe redis-backup` or `velero backup logs redis-backup` for more details.
[root@master1 /root]# velero backup get
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR
demo Completed 0 0 2024-11-21 06:46:30 +0000 UTC 29d default <none>
redis-backup Completed 0 0 2024-11-21 08:01:48 +0000 UTC 29d default <none>
[root@master1 /root]# velero backup describe redis-backup --details
Name: redis-backup
Namespace: velero
Labels: velero.io/storage-location=default
Annotations: velero.io/resource-timeout=10m0svelero.io/source-cluster-k8s-gitversion=v1.29.2velero.io/source-cluster-k8s-major-version=1velero.io/source-cluster-k8s-minor-version=29Phase: CompletedNamespaces:Included: demoExcluded: <none>Resources:Included: *Excluded: <none>Cluster-scoped: autoLabel selector: <none>Or label selector: <none>Storage Location: defaultVelero-Native Snapshot PVs: auto
Snapshot Move Data: false
Data Mover: veleroTTL: 720h0m0sCSISnapshotTimeout: 10m0s
ItemOperationTimeout: 4h0m0sHooks: <none>Backup Format Version: 1.1.0Started: 2024-11-21 08:01:48 +0000 UTC
Completed: 2024-11-21 08:01:52 +0000 UTCExpiration: 2024-12-20 08:01:48 +0000 UTCTotal items to be backed up: 23
Items backed up: 23Resource List:apiextensions.k8s.io/v1/CustomResourceDefinition:- ciliumendpoints.cilium.ioapps/v1/Deployment:- demo/demoappapps/v1/ReplicaSet:- demo/demoapp-7c58cd6bbcilium.io/v2/CiliumEndpoint:- demo/demoapp-7c58cd6bb-cs2fc- demo/demoapp-7c58cd6bb-r4rcb- demo/demoapp-7c58cd6bb-w7ckj- demo/redis-with-nfs-pvcv1/ConfigMap:- demo/kube-root-ca.crtv1/Event:- demo/nfs-pvc.17beb8890487bc71- demo/nfs-pvc.17beb88905411edd- demo/nfs-pvc.17beb8890aa62b1b- demo/redis-with-nfs-pvc.17beb88c587fdc81- demo/redis-with-nfs-pvc.17beb88c78733a92- demo/redis-with-nfs-pvc.17beb88c78e45daf- demo/redis-with-nfs-pvc.17beb88c7abb19b1v1/Namespace:- demov1/PersistentVolume:- pvc-6a17223d-6628-4eb4-877c-954e9b0eef09v1/PersistentVolumeClaim:- demo/nfs-pvcv1/Pod:- demo/demoapp-7c58cd6bb-cs2fc- demo/demoapp-7c58cd6bb-r4rcb- demo/demoapp-7c58cd6bb-w7ckj- demo/redis-with-nfs-pvcv1/ServiceAccount:- demo/defaultBackup Volumes:Velero-Native Snapshots: <none included>CSI Snapshots: <none included>Pod Volume Backups - kopia:Completed:demo/redis-with-nfs-pvc: data-storageHooksAttempted: 0
HooksFailed: 0#################这里一定要注意Phase的状态，一定是Completed，否则数据会不完整，这里需要多等待状态完成#################【5】模拟故障 删除ns ：demo
[root@master1 /root]# kubectl delete ns demo
namespace "demo" deleted
[root@master1 /root]# kubectl get pv
No resources found
[root@master1 /root]# velero backup get
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR
demo Completed 0 0 2024-11-21 06:46:30 +0000 UTC 29d default <none>
redis-backup Completed 0 0 2024-11-21 08:01:48 +0000 UTC 29d default <none>
[root@master1 /root]# velero restore create --from-backup redis-backup
Restore request "redis-backup-20240321080651" submitted successfully.
Run `velero restore describe redis-backup-20240321080651` or `velero restore logs redis-backup-20240321080651` for more details.
[root@master1 /root]# kubectl get pvc
No resources found in default namespace.
root@ubuntu-k8s-master01:~# kubectl get pvc -n demo
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
nfs-pvc Bound pvc-5628dd5b-5360-4e33-93b6-8ce42e2da07e 3Gi RWX nfs-csi <unset> 9s
[root@master1 /root]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
pvc-5628dd5b-5360-4e33-93b6-8ce42e2da07e 3Gi RWX Delete Bound demo/nfs-pvc nfs-csi <unset> 24s
[root@master1 /root]# kubectl get pods -n demo
NAME READY STATUS RESTARTS AGE
redis-with-nfs-pvc 1/1 Running 0 2m13s#【6】-验证数据恢复情况
#NFS 服务确认
[root@k8s-harbor ~]# cd /data/nfs/
[root@k8s-harbor nfs]# ll
total 0
drwxr-xr-x 3 polkitd root 74 Mar 21 16:08 pvc-5628dd5b-5360-4e33-93b6-8ce42e2da07e#服务中数据确认
[root@master1 /root]# kubectl exec -it redis-with-nfs-pvc -n demo -- /bin/sh
Defaulted container "redis" out of: redis, restore-wait (init)
/data # ls
BGSAVE dump.rdb exit set
/data # redis-cli
127.0.0.1:6379> get mykey
"BIRKHOFF 2024-11-21"
10. 总结
Velero 是一个强大的 Kubernetes 集群备份和恢复工具，它提供了细粒度的备份选项，支持多种存储解决方案，并允许跨集群迁移。与 ETCD 快照备份相比，Velero 提供了更多的灵活性和选择性，也让他成为 Kubernetes 集群数据保护的理想选择。