【二进制部署k8s-1.29.4】二、证书及配置文件启动脚步的准备

简介

本章节主要准备二进制安装k8s的过程中所使用到的证书配置文件，怎样生成证书，以及etcd、master端组件、worker端组件所用到的配置文件和启动脚本，同时利用脚本生成证书、和生成kubecofig配置文件。

一.准备证书相关的配置文件

1.1.ca-config.json

定义ca证书的过期时间

{"signing": {"default": {"expiry": "175200h"},"profiles": {"kubernetes": {"expiry": "175200h","usages": ["signing","key encipherment","server auth","client auth"]}}}}

1.2.ca-csr.json

定义ca证书的加密算法、地域及组织单位

{"CN": "kubernetes","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","L": "Guangzhou","ST": "Guangdong","O": "k8s","OU": "System"}]
}

1.3.etcd-csr.json

定义etcd证书中的域名、IP、加密算法及组织单位,配置中的三个IP为安装etcd的IP，现在是将etcd安装在master的三个IP上，所以配置的是master的IP

{"CN": "etcd","hosts": ["10.16.120.81","10.16.120.82","10.16.120.83","127.0.0.1"],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","L": "Guangzhou","ST": "Guangdong"}]
}

1.4.kube-apiserver-csr.json

定义api-server证书中的域名、IP、加密算法及组织单位，配置中的IP主要是master的IP，以及配置api-server的vip，或调用api-server的域名

{"CN": "kubernetes","hosts": ["127.0.0.1","10.16.120.80","10.16.120.81","10.16.120.82","10.16.120.83","10.1.0.1","yt-pcauto-k8s.pc.com.cn","kubernetes","kubernetes.default","kubernetes.default.svc","kubernetes.default.svc.cluster","kubernetes.default.svc.cluster.local"],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","L": "Guangzhou","ST": "Guangdong","O": "k8s","OU": "system"}]
}

1.5.kube-controller-manager-csr.json

定义kube-controller-manager 证书中的api证书地址、节点IP、加密算法及组织单位，配置中的IP是kube-apiserver的vip，域名或127.0.0.1，主要是controller-manager一般都是和apiserver安装在同样的机器上

{"CN": "system:kube-controller-manager","hosts": ["127.0.0.1","10.16.120.80","yt-pcauto-k8s.pc.com.cn"],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Guangdong","L": "Guangzhou","O": "system:kube-controller-manager","OU": "system"}]
}

1.6.kube-scheduler-csr.json

定义kube-scheduler证书中的api证书地址、节点IP、加密算法及组织单位，配置中的IP是kube-apiserver的vip，域名或127.0.0.1，主要是controller-manager一般都是和apiserver安装在同样的机器上

{"CN": "system:kube-scheduler","hosts": ["127.0.0.1","10.16.120.80","yt-pcauto-k8s.pc.com.cn"],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Guangdong","L": "Guangzhou","O": "system:kube-scheduler","OU": "system"}]
}

1.7.admin-csr.json

该配置是用于生成k8s管理客户端kubectl所需的kubeconfig时需要公钥和私钥所必须的证书配置文件

{"CN": "admin","hosts": [],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Guangdong","L": "Guangzhou","O": "system:masters",             "OU": "system"}]
}

1.8.proxy-client-csr.json

kube-apiserver 的另一种访问方式就是使用 kubectl proxy 来代理访问, 而该证书就是用来支持SSL代理访问的. 在该种访问模式下, 我们是以http的方式发起请求到代理服务的, 此时, 代理服务会将该请求发送给 kube-apiserver, 在此之前, 代理会将发送给 kube-apiserver 的请求头里加入证书信息

{"CN": "aggregator","hosts": [],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Guangdong","L": "Guangzhou","O": "system:masters","OU": "System"}]}

二.安装客户端相关软件及命令

该步骤主要是将部署过程中需要用的一些命令先进行安装，主要就是将以下二进制可执行命令拷贝到/usr/bin目录，该部分软件可以在其中一台master机器上进行安装，也可以在独立的机器上进行安装。

软件	用途
cfssl，cfssl-certinfo，cfssljson	用于生成安装所需的证书
cilium	用于查看cilium的安装状及卸载cilium的客户端
helm	用于安装charts的客户端，例如安装cilium,安装credn，安装ingress等
kubectl,kubectl-convert	k8s客户端软件，kubectl是管理k8s必需的的客户端软件

三.生成证书

将第一步所有的的配置文件放在csr-conf这样一个目录下，然后执行以下脚本生成证书

#!/bin/shetcd_cert_dir="install_etcd"  #存放etcd证书的目录
master_cert_dir="install_master/cert" #存放安装master所需证书的目录[ -d $master_cert_dir ] || mkdir -p $master_cert_dir
[ -d $etcd_cert_dir ] || mkdir -p $etcd_cert_dir
[ -d client ] || mkdir -p client #client目录用于存放生成kubectl命令的配置及相关证书
[ -d ca ] || mkdir -p ca    #存放ca证书及私钥echo "create ca.pem ca-key.pem======="
cfssl gencert -initca csr-conf/ca-csr.json | cfssljson -bare ca -
mv ca.pem ca-key.pem ca/
rm ca.csrecho "create etcd.pem etcd-key.pem======="
cfssl gencert -ca=ca/ca.pem -ca-key=ca/ca-key.pem -config=csr-conf/ca-config.json -profile=kubernetes csr-conf/etcd-csr.json | cfssljson -bare $etcd_cert_dir/etcd
rm -f $etcd_cert_dir/etcd.csrecho "create kube-apiserver.pem kube-apiserver-key.pem======="
cfssl gencert -ca=ca/ca.pem -ca-key=ca/ca-key.pem -config=csr-conf/ca-config.json -profile=kubernetes csr-conf/kube-apiserver-csr.json | cfssljson -bare $master_cert_dir/kube-apiserver
rm -f $master_cert_dir/kube-apiserver.csrecho "create kube-scheduler.pem kube-scheduler-key.pem======="
cfssl gencert -ca=ca/ca.pem -ca-key=ca/ca-key.pem -config=csr-conf/ca-config.json -profile=kubernetes csr-conf/kube-scheduler-csr.json | cfssljson -bare $master_cert_dir/kube-scheduler
rm -f $master_cert_dir/kube-scheduler.csrecho "create kube-controller-manager.pem kube-controller-manager-key.pem======="
cfssl gencert -ca=ca/ca.pem -ca-key=ca/ca-key.pem -config=csr-conf/ca-config.json -profile=kubernetes csr-conf/kube-controller-manager-csr.json | cfssljson -bare $master_cert_dir/kube-controller-manager
rm -f $master_cert_dir/kube-controller-manager.csrecho "create proxy-client.pem proxy-client-key.pem======="
cfssl gencert -ca=ca/ca.pem -ca-key=ca/ca-key.pem -config=csr-conf/ca-config.json -profile=kubernetes csr-conf/proxy-client-csr.json  | cfssljson -bare $master_cert_dir/proxy-client
rm -f $master_cert_dir/proxy-client.csrecho "create admin.pem admin-key.pem======="
cfssl gencert -ca=ca/ca.pem -ca-key=ca/ca-key.pem -config=csr-conf/ca-config.json -profile=kubernetes csr-conf/admin-csr.json | cfssljson -bare client/admin
rm -fv client/admin.csr

四.准备k8s配置文件

3.1.etcd.conf

10.16.120.81 的配置，每台机不一样

#[Member]
ETCD_NAME="etcd01"
ETCD_DATA_DIR="/opt/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://10.16.120.81:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.16.120.81:2379,http://127.0.0.1:2379"#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.16.120.81:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.16.120.81:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://10.16.120.81:2380,etcd02=https://10.16.120.82:2380,etcd03=https://10.16.120.83:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_LISTEN_METRICS_URLS="http://0.0.0.0:2381"

10.16.120.82 的配置，每台机不一样

#[Member]
ETCD_NAME="etcd02"
ETCD_DATA_DIR="/opt/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://10.16.120.82:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.16.120.82:2379,http://127.0.0.1:2379"#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.16.120.82:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.16.120.82:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://10.16.120.81:2380,etcd02=https://10.16.120.82:2380,etcd03=https://10.16.120.83:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_LISTEN_METRICS_URLS="http://0.0.0.0:2381"

10.16.120.83 的配置，每台机不一样

#[Member]
ETCD_NAME="etcd03"
ETCD_DATA_DIR="/opt/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://10.16.120.83:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.16.120.83:2379,http://127.0.0.1:2379"#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.16.120.83:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.16.120.83:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://10.16.120.81:2380,etcd02=https://10.16.120.82:2380,etcd03=https://10.16.120.83:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_LISTEN_METRICS_URLS="http://0.0.0.0:2381"

3.2.kube-apiserver.conf

注意配置中的文件、证书路径。需要修改的地方主要就是etcd的IP,配置中的pem证书文件是在“三.生成证书” 时生成的，其中的token.csv 会在“五.准备kubeconfig配置文件”中生成

KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \--anonymous-auth=false \--secure-port=6443 \--authorization-mode=Node,RBAC \--runtime-config=api/all=true \--enable-bootstrap-token-auth \--service-cluster-ip-range=10.1.0.0/16 \--token-auth-file=/opt/kubernetes/conf/token.csv \--service-node-port-range=30000-50000 \--tls-cert-file=/opt/kubernetes/ssl/kube-apiserver.pem \--tls-private-key-file=/opt/kubernetes/ssl/kube-apiserver-key.pem \--client-ca-file=/opt/kubernetes/ssl/ca.pem \--kubelet-client-certificate=/opt/kubernetes/ssl/kube-apiserver.pem \--kubelet-client-key=/opt/kubernetes/ssl/kube-apiserver-key.pem \--kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS \--service-account-issuer=https://kubernetes.default.svc \--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \--service-account-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \--etcd-cafile=/opt/etcd/ssl/ca.pem \--etcd-certfile=/opt/etcd/ssl/etcd.pem \--etcd-keyfile=/opt/etcd/ssl/etcd-key.pem \--etcd-servers=https://10.16.120.81:2379,https://10.16.120.82:2379,https://10.16.120.83:2379 \--allow-privileged=true \--audit-log-maxage=5 \--audit-log-maxbackup=3 \--audit-log-maxsize=100 \--audit-log-path=/opt/kubernetes/logs/kube-apiserver-audit.log \--requestheader-allowed-names=aggregator \--requestheader-group-headers=X-Remote-Group \--requestheader-username-headers=X-Remote-User \--requestheader-extra-headers-prefix=X-Remote-Extra- \--requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \--proxy-client-cert-file=/opt/kubernetes/ssl/proxy-client.pem \--proxy-client-key-file=/opt/kubernetes/ssl/proxy-client-key.pem \--v=4"

3.3.kube-controller-manager.conf

注意配置中的文件、证书路径，以及service和pod的网段，kubeconfig会在“五.准备kubeconfig配置文件”中生成

KUBE_CONTROLLER_MANAGER_OPTS="--v=2 \--kubeconfig=/opt/kubernetes/conf/kube-controller-manager.kubeconfig \--horizontal-pod-autoscaler-sync-period=10s \--service-cluster-ip-range=10.1.0.0/16 \--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \--allocate-node-cidrs=true \--cluster-cidr=10.2.0.0/16 \--cluster-signing-duration=175200h \--root-ca-file=/opt/kubernetes/ssl/ca.pem \--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \--leader-elect=true \--feature-gates=RotateKubeletServerCertificate=true \--controllers=*,bootstrapsigner,tokencleaner \--tls-cert-file=/opt/kubernetes/ssl/kube-controller-manager.pem \--tls-private-key-file=/opt/kubernetes/ssl/kube-controller-manager-key.pem \--use-service-account-credentials=true"

3.4.kube-scheduler.conf

注意配置中的文件路径，kueconfig会在“五.准备kubeconfig配置文件”中生成

KUBE_SCHEDULER_OPTS="--kubeconfig=/opt/kubernetes/conf/kube-scheduler.kubeconfig \
--leader-elect=true \
--v=2"

3.5.kubelet.yaml

其中10.1.0.2是安装conredns的IP，提前定义好次IP。/opt/kubernetes/ssl/ca.pem 为ca的证书路径，/run/systemd/resolve/resolv.conf为系统的resolved的dns配置路径，不配置此项会导致读取/etc/resolv.conf,而/etc/resolv.conf是/run/systemd/resolve/stub-resolv.conf的软连接，里面配置了本地缓存dns,127.0.0.1:53,会和k8s导致dns冲突

kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: systemd 
clusterDNS:
- 10.1.0.2
clusterDomain: cluster.local 
failSwapOn: false
authentication:anonymous:enabled: falsewebhook:cacheTTL: 2m0senabled: truex509:clientCAFile: /opt/kubernetes/ssl/ca.pem 
authorization:mode: Webhookwebhook:cacheAuthorizedTTL: 5m0scacheUnauthorizedTTL: 30s
evictionHard:imagefs.available: 15%memory.available: 100Minodefs.available: 10%nodefs.inodesFree: 5%
maxOpenFiles: 2048000
maxPods: 200
resolvConf: /run/systemd/resolve/resolv.conf

3.6.containerd配置文件

containerd的配置文件，需要在worker上安装好containerd时,然后再执行containerd命令导出默认配置，并修改里面的镜像地址，也可以解压containerd的安装包，拷贝containerd的执行文件出来执行导出配置文件。

containerd config default | sudo tee /etc/containerd/config.toml
sed -i 's#SystemdCgroup.*#SystemdCgroup = true#' /etc/containerd/config.toml
sed -i 's#sandbox_image.*#sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.8"#' /etc/containerd/config.toml

五.生成kubeconfig配置文件

kubeconfig的配置文件在安装kube-control-manger,kube-schedul,kubelet,以及配置kubectl客户端时都需要用到。该脚本中所使用到的路径与“三.生成证书”中所使用的的路径一致，如果路径有变动，需要两个脚本都修改一下存放路径。

#!/bin/bashca_dir="ca"  #存放ca证书的路径，与第二步生成证书时的路径一致
token_dir="install_master" #存放token.csv的路径
CONFIG_DIR="install_master/kubeconfig" #存放master端使用到kubeconfig的保存路径
worker_dir="install_worker/config"   #存放worker端使用到kubeconfig的保存路径
master_cert_dir="install_master/cert"   #存放maser端使用到的证书的路径，与第二步生成证书时的路径一致
client_dir="client" ##存放client生成的kubecofig以及client端的证书，与第二步生成证书时的路径一致KUBE_APISERVER="https://yt-pcauto-k8s.pc.com.cn:6443"   #apiserver的地址[ -d $worker_dir ] || mkdir -p $worker_direcho "create token ====="
cat > $token_dir/token.csv << EOF
$(head -c 16 /dev/urandom | od -An -t x | tr -d ' '),kubelet-bootstrap,10001,"system:bootstrappers"
EOFecho "create kube-controller-manager.kubeconfig ====="
kubectl config set-cluster kubernetes \--certificate-authority=$ca_dir/ca.pem \--embed-certs=true \--server=${KUBE_APISERVER} \--kubeconfig=$CONFIG_DIR/kube-controller-manager.kubeconfigkubectl config set-credentials system:kube-controller-manager \--client-certificate=$master_cert_dir/kube-controller-manager.pem \--client-key=$master_cert_dir/kube-controller-manager-key.pem \--embed-certs=true \--kubeconfig=$CONFIG_DIR/kube-controller-manager.kubeconfigkubectl config set-context system:kube-controller-manager \--cluster=kubernetes \--user=system:kube-controller-manager \--kubeconfig=$CONFIG_DIR/kube-controller-manager.kubeconfigkubectl config use-context system:kube-controller-manager --kubeconfig=$CONFIG_DIR/kube-controller-manager.kubeconfigecho "create kube-scheduler.kubeconfig ====="
kubectl config set-cluster kubernetes \--certificate-authority=$ca_dir/ca.pem \--embed-certs=true \--server=${KUBE_APISERVER} \--kubeconfig=$CONFIG_DIR/kube-scheduler.kubeconfigkubectl config set-credentials system:kube-scheduler \--client-certificate=$master_cert_dir/kube-scheduler.pem \--client-key=$master_cert_dir/kube-scheduler-key.pem \--embed-certs=true \--kubeconfig=$CONFIG_DIR/kube-scheduler.kubeconfigkubectl config set-context system:kube-scheduler \--cluster=kubernetes \--user=system:kube-scheduler \--kubeconfig=$CONFIG_DIR/kube-scheduler.kubeconfigkubectl config use-context system:kube-scheduler --kubeconfig=$CONFIG_DIR/kube-scheduler.kubeconfigecho "create kubelet-bootstrap.kubeconfig ====="
TOKEN=$(awk -F "," '{print $1}' $token_dir/token.csv)
kubectl config set-cluster kubernetes \--certificate-authority=$ca_dir/ca.pem \--embed-certs=true \--server=${KUBE_APISERVER} \--kubeconfig=$worker_dir/kubelet-bootstrap.kubeconfigkubectl config set-credentials kubelet-bootstrap \--token=${TOKEN} \--kubeconfig=$worker_dir/kubelet-bootstrap.kubeconfigkubectl config set-context default \--cluster=kubernetes \--user=kubelet-bootstrap \--kubeconfig=$worker_dir/kubelet-bootstrap.kubeconfigkubectl config use-context default --kubeconfig=$worker_dir/kubelet-bootstrap.kubeconfigecho "create client kube.config ====="
kubectl config set-cluster kubernetes \--certificate-authority=$ca_dir/ca.pem \--embed-certs=true \--server=${KUBE_APISERVER} \--kubeconfig=$client_dir/kube.kubeconfigkubectl config set-credentials admin \--client-certificate=$client_dir/admin.pem \--client-key=$client_dir/admin-key.pem \--embed-certs=true \--kubeconfig=$client_dir/kube.kubeconfigkubectl config set-context kubernetes \--cluster=kubernetes \--user=admin \--kubeconfig=$client_dir/kube.kubeconfigkubectl config use-context kubernetes --kubeconfig=$client_dir/kube.kubeconfig

六.准备启动脚本

6.1.etcd.service

etcd的启动脚本

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target[Service]
Type=notify
EnvironmentFile=-/opt/etcd/conf/etcd.conf
WorkingDirectory=/opt/etcd/
ExecStart=/opt/etcd/bin/etcd \--cert-file=/opt/etcd/ssl/etcd.pem \--key-file=/opt/etcd/ssl/etcd-key.pem \--trusted-ca-file=/opt/etcd/ssl/ca.pem \--peer-cert-file=/opt/etcd/ssl/etcd.pem \--peer-key-file=/opt/etcd/ssl/etcd-key.pem \--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \--peer-client-cert-auth \--client-cert-auth
Restart=on-failure
RestartSec=5
LimitNOFILE=65536[Install]
WantedBy=multi-user.target

6.2.kube-apiserver.service

kube-apiserver的启动脚本

[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=etcd.service
Wants=etcd.service[Service]
EnvironmentFile=-/opt/kubernetes/conf/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536[Install]
WantedBy=multi-user.target

6.3.kube-controller-manager.service

kube-controller-manager的启动脚本

[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes[Service]
EnvironmentFile=-/opt/kubernetes/conf/kube-controller-manager.conf
ExecStart=/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
RestartSec=5[Install]
WantedBy=multi-user.target

6.4.kube-scheduler.service

kube-scheduler的启动脚本

[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes[Service]
EnvironmentFile=-/opt/kubernetes/conf/kube-scheduler.conf
ExecStart=/opt/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure
RestartSec=5[Install]
WantedBy=multi-user.target

6.5.kubelet.service

worker端kubelet的启动脚本

[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=containerd.service
Requires=containerd.service[Service]
ExecStart=/opt/kubernetes/bin/kubelet \--hostname-override=node-hostname \ #此处需要配置正确的节点的主机名--bootstrap-kubeconfig=/opt/kubernetes/conf/kubelet-bootstrap.kubeconfig \--cert-dir=/opt/kubernetes/ssl \--client-ca-file=/opt/kubernetes/ssl/ca.pem \--kubeconfig=/opt/kubernetes/conf/kubelet.kubeconfig \--config=/opt/kubernetes/conf/kubelet.yaml \--container-runtime-endpoint=unix:///var/run/containerd/containerd.sock \--v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536[Install]
WantedBy=multi-user.target

6.6.containerd启动脚本

在安装containerd时，解压cri-containerd-1.7.16-linux-amd64.tar.gz, tar zxvf cri-containerd-1.7.16-linux-amd64.tar.gz -C /就会在/etc/systemd/system/containerd.service 路径下有启动脚本

七.总结

建议将以上文件生成后统一放到一个目录，例如放到install_k8s的目录，然后将下载的软件也放在此目录，将生成证书的脚本和生成kubeconfig的脚本放在install_k8s目录下，在生成证书、配置文件、启动脚本以后，方便后边的安装步骤找对应的文件。