zxc的自留地

About Me

搭建k8s集群

December 13, 2022

前言

安装

0.0 set hostname

Role IP
k8s-master 10.0.0.175
k8s-node1 10.0.0.158

准备两台内网互通的机器，

echo "
10.0.0.175 k8s-master
10.0.0.158 k8s-node1
" >> /etc/hosts

01.kubeadm install

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt update -y
sudo apt -y install vim git curl wget

# 安装不同的版本
sudo apt -y isntall kubelet=1.24.3-00 kubeadm=1.24.3-00 kubectl=1.24.3-00
sudo apt -y install kubelet=1.25.4-00 kubeadm=1.25.4-00 kubectl=1.25.4-00

02.linux config

echo "memory swapoff"
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
sudo swapoff -a
sysctl net.ipv4.conf.all.forwarding=1
sysctl --system

03.containerd install and config


# 手动安装
wget https://github.com/containerd/containerd/releases/download/v1.6.2/containerd-1.6.2-linux-arm64.tar.gz
sudo tar Czxvf /usr/local containerd-1.6.2-linux-arm64.tar.gz


wget https://raw.githubusercontent.com/containerd/containerd/main/containerd.service
sudo mv containerd.service /usr/lib/systemd/system/

wget https://github.com/opencontainers/runc/releases/download/v1.1.1/runc.amd64
sudo install -m 755 runc.amd64 /usr/local/sbin/runc

# 通过依赖安装
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list
sudo apt update
sudo apt install -y containerd.io


# 配置containerd
mkdir -p /etc/containerd
containerd config default>/etc/containerd/config.toml
sudo sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml
sudo systemctl restart containerd
sudo systemctl enable containerd

cat /etc/crictl.yaml << EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: true
EOF

04.k8s config

sudo tee /etc/modules-load.d/containerd.conf <<EOF
overlay
br_netfilter
EOF

sysctl --system
sudo tee /etc/modules-load.d/containerd.conf <<EOF
overlay
br_netfilter
EOF
sysctl --system

05.kubeadm 相关操作

init

kubeadm init --apiserver-advertise-address=10.0.0.175 \ # master的地址
             --kubernetes-version v1.25.4 \ # k8s版本
             --pod-network-cidr=10.244.0.0/16 \ pod ip池
             --cri-socket=unix:///run/containerd/containerd.sock \ # cri sock 路径
             --v=5

cni

安装 cni 之前 kubectl get node 都是 NotReady 的，安装完 cni 就是 Ready 了

curl https://docs.projectcalico.org/manifests/calico.yaml -O
KUBECONFIG=/etc/kubernetes/admin.conf kubectl apply -f calico.yaml

join

如果一切顺利的话，就可以把其他节点加入集群了，其他节点需要执行前 5 步(00-04)

kubeadm join 10.0.0.175:6443 --token emuni5.vpyzqfm51d2dpk6y --discovery-token-ca-cert-hash sha256:ad547694790f288d2e801d3d933723a472b73f6809619c9b13a9e56784203d0d

# 由于token会过期，可以通过命令重新生成
kubeadm token create --print-join-command

clean

sudo kubeadm reset --cri-socket=unix:///run/containerd/containerd.sock -f
sudo rm -rf /var/lib/{calico,etcd,kubelet,kubernetes,cni} /etc/cni/net.d /etc/kubernetes /opt/cni/bin/*

测试

命令补全

# 临时生效
source <(crictl completion)
source <(kubeadm completion bash)
source <(kubectl completion bash)

# 永久生效
crictl completion > /etc/bash_completion.d/crictl
kubectl completion bash > /etc/bash_completion.d/kubectl
kubeadm completion bash > /etc/bash_completion.d/kubeadm

创建 deployment

kubectl create deployment nginx-app --image=nginx --replicas=2
kubectl expose deployment nginx-app --type=NodePort --port=80

开放公网访问

当节点有公网 IP，想要通过公网 IP 访问时，需要修改 certSANs 的信息 kubectl -n kube-system get configmap kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' > kubeadm.yaml

  1. 生成 kubeadm.yaml
kubectl -n kube-system get configmap kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' > kubeadm.yaml
  1. 修改 kubeadm.yaml，添加 ip
apiServer:
  certSANs:
  - "172.29.50.162"
  - "k8s.domain.com"
  - "other-k8s.domain.net"
  extraArgs:
    authorization-mode: Node,RBAC
  timeoutForControlPlane: 4m0s
  1. 生成证书和更新配置
mv /etc/kubernetes/pki/apiserver.{crt,key} ~
kubeadm init phase certs apiserver --config kubeadm.yaml
kubeadm init phase upload-config kubeadm --config kubeadm.yaml
  1. 使用原来的 kubeconfig 即可

Reference