使用kubeadm在GCE上构建一个包含1个控制平面和3个工作节点的Kubernetes集群
准备一个GCE实例作为控制平面,三个GCE实例作为工作节点,然后使用kubeadm构建Kubernetes集群。
同时,选择containerd作为容器运行时。
在本地机器上使用了 fish shell,在 SSH 到 GCE 实例上使用了 bash shell。为了避免混淆,在代码块的标题中标明了使用的shell名称。
所有版本
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:09:38Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
$ containerd -v
containerd github.com/containerd/containerd 1.3.3-0ubuntu2.2
$ kubelet --version
Kubernetes v1.20.4
$ kubectl version --client
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:12:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
$ k get po -A -o jsonpath="{..image}" --kubeconfig ./admin.conf |\
tr -s '[[:space:]]' '\n' |\
sort |\
uniq -c
16 docker.io/calico/cni:v3.18.0
2 docker.io/calico/kube-controllers:v3.18.0
8 docker.io/calico/node:v3.18.0
8 docker.io/calico/pod2daemon-flexvol:v3.18.0
3 docker.io/library/nginx:latest
4 k8s.gcr.io/coredns:1.7.0
2 k8s.gcr.io/etcd:3.4.13-0
2 k8s.gcr.io/kube-apiserver:v1.20.4
2 k8s.gcr.io/kube-controller-manager:v1.20.4
8 k8s.gcr.io/kube-proxy:v1.20.4
2 k8s.gcr.io/kube-scheduler:v1.20.4
创建计算资源
我们将参考《Kubernetes the Hard Way》中的“Provisioning Compute Resources”和Calico在Google Compute Engine(GCE)上的“Self-managed Kubernetes”来进行操作。
创建VPC
创建一个VPC来创建集群。
$ gcloud compute networks create kubernetes-by-kubeadm --subnet-mode custom
Created [https://www.googleapis.com/compute/v1/projects/sandbox-project/global/networks/kubernetes-by-kubeadm].
NAME SUBNET_MODE BGP_ROUTING_MODE IPV4_RANGE GATEWAY_IPV4
kubernetes-by-kubeadm CUSTOM REGIONAL
$ gcloud compute networks subnets create kubernetes \
--network kubernetes-by-kubeadm \
--range 10.240.0.0/24
Created [https://www.googleapis.com/compute/v1/projects/sandbox-project/regions/asia-northeast1/subnetworks/kubernetes].
NAME REGION NETWORK RANGE
kubernetes asia-northeast1 kubernetes-by-kubeadm 10.240.0.0/24
创建防火墙规则
集群内部允许 tcp、udp、icmp、ipip 的通信,而从集群外部允许 tcp:22、tcp:6443、icmp 的通信。
$ gcloud compute firewall-rules create kubernetes-by-kubeadm-allow-internal \
--allow tcp,udp,icmp,ipip \
--network kubernetes-by-kubeadm \
--source-ranges 10.240.0.0/24,10.200.0.0/16
Creating firewall...⠹Created [https://www.googleapis.com/compute/v1/projects/sandbox-project/global/firewalls/kubernetes-by-kubeadm-allow-internal].
Creating firewall...done.
NAME NETWORK DIRECTION PRIORITY ALLOW DENY DISABLED
kubernetes-by-kubeadm-allow-internal kubernetes-by-kubeadm INGRESS 1000 tcp,udp,icmp,ipip False
$ gcloud compute firewall-rules create kubernetes-by-kubeadm-allow-external \
--allow tcp:22,tcp:6443,icmp \
--network kubernetes-by-kubeadm \
--source-ranges 0.0.0.0/0
Creating firewall...⠹Created [https://www.googleapis.com/compute/v1/projects/sandbox-project/global/firewalls/kubernetes-by-kubeadm-allow-external].
Creating firewall...done.
NAME NETWORK DIRECTION PRIORITY ALLOW DENY DISABLED
kubernetes-by-kubeadm-allow-external kubernetes-by-kubeadm INGRESS 1000 tcp:22,tcp:443,tcp:6443,icmp False
分配公共 IP
由於這次只有一個控制平面,我們可以將為控制平面準備的實例的外部IP直接用作API終端點,但考慮到實現高可用架構並進行負載平衡,我們應該預先分配一個公有IP地址。
$ gcloud compute addresses create kubernetes-by-kubeadm \
--region (gcloud config get-value compute/region)
Created [https://www.googleapis.com/compute/v1/projects/sandbox-project/regions/asia-northeast1/addresses/kubernetes-by-kubeadm].
$ gcloud compute addresses list
NAME ADDRESS/RANGE TYPE PURPOSE NETWORK REGION SUBNET STATUS
kubernetes-by-kubeadm 34.85.15.20 EXTERNAL asia-northeast1 RESERVED
创建计算实例
首先是用于控制平面的实例。
$ gcloud compute instances create controller \
--async \
--boot-disk-size 200GB \
--can-ip-forward \
--image-family ubuntu-2004-lts \
--image-project ubuntu-os-cloud \
--machine-type n1-standard-4 \
--private-network-ip 10.240.0.10 \
--scopes compute-rw,storage-ro,service-management,service-control,logging-write,monitoring \
--subnet kubernetes \
--tags kubernetes-by-kubeadm,controller
NOTE: The users will be charged for public IPs when VMs are created.
Instance creation in progress for [controller]: https://www.googleapis.com/compute/v1/projects/sandbox-project/zones/asia-northeast1-a/operations/operation-1613193837534-5bb30f5a3dc98-ba859467-b3c39485
Use [gcloud compute operations describe URI] command to check the status of the operation(s).
接下来,创建三个用于工作节点的实例。
$ for i in 0 1 2
gcloud compute instances create worker-{$i} \
--async \
--boot-disk-size 200GB \
--can-ip-forward \
--image-family ubuntu-2004-lts \
--image-project ubuntu-os-cloud \
--machine-type n1-standard-4 \
--private-network-ip 10.240.0.2{$i} \
--scopes compute-rw,storage-ro,service-management,service-control,logging-write,monitoring \
--subnet kubernetes \
--tags kubernetes-by-kubeadm,worker
end
NOTE: The users will be charged for public IPs when VMs are created.
Instance creation in progress for [worker-0]: https://www.googleapis.com/compute/v1/projects/sandbox-project/zones/asia-northeast1-a/operations/operation-1613193959993-5bb30fcf070d9-aa954478-17a67d14
Use [gcloud compute operations describe URI] command to check the status of the operation(s).
NOTE: The users will be charged for public IPs when VMs are created.
Instance creation in progress for [worker-1]: https://www.googleapis.com/compute/v1/projects/sandbox-project/zones/asia-northeast1-a/operations/operation-1613193964271-5bb30fd31b636-5bc6d372-19d8209d
Use [gcloud compute operations describe URI] command to check the status of the operation(s).
NOTE: The users will be charged for public IPs when VMs are created.
Instance creation in progress for [worker-2]: https://www.googleapis.com/compute/v1/projects/sandbox-project/zones/asia-northeast1-a/operations/operation-1613193968163-5bb30fd6d1b22-133c16bc-8135db2a
Use [gcloud compute operations describe URI] command to check the status of the operation(s).
外部负载均衡器的配置
在这个时候,我们会对刚才分配的公共IP地址进行负载均衡(LB)的设置。
我们会进行健康检查的设置,并为目标池设置用于控制平面的实例。
$ set KUBERNETES_PUBLIC_ADDRESS (gcloud compute addresses describe kubernetes-by-kubeadm \
--region (gcloud config get-value compute/region) \
--format 'value(address)')
$ gcloud compute http-health-checks create kubernetes \
--description "Kubernetes Health Check" \
--host "kubernetes.default.svc.cluster.local" \
--request-path "/healthz"
Created [https://www.googleapis.com/compute/v1/projects/sandbox-project/global/httpHealthChecks/kubernetes].
NAME HOST PORT REQUEST_PATH
kubernetes kubernetes.default.svc.cluster.local 80 /healthz
$ gcloud compute firewall-rules create kubernetes-by-kubeadm-allow-health-check \
--network kubernetes-by-kubeadm \
--source-ranges 209.85.152.0/22,209.85.204.0/22,35.191.0.0/16 \
--allow tcp
Creating firewall...⠹Created [https://www.googleapis.com/compute/v1/projects/sandbox-project/global/firewalls/kubernetes-by-kubeadm-allow-health-check].
Creating firewall...done.
NAME NETWORK DIRECTION PRIORITY ALLOW DENY DISABLED
kubernetes-by-kubeadm-allow-health-check kubernetes-by-kubeadm INGRESS 1000 tcp False
$ gcloud compute target-pools create kubernetes-target-pool \
--http-health-check kubernetes
Created [https://www.googleapis.com/compute/v1/projects/sandbox-project/regions/asia-northeast1/targetPools/kubernetes-target-pool].
NAME REGION SESSION_AFFINITY BACKUP HEALTH_CHECKS
kubernetes-target-pool asia-northeast1 NONE kubernetes
$ gcloud compute target-pools add-instances kubernetes-target-pool \
--instances controller
Updated [https://www.googleapis.com/compute/v1/projects/sandbox-project/regions/asia-northeast1/targetPools/kubernetes-target-pool].
$ gcloud compute forwarding-rules create kubernetes-forwarding-rule \
--address $KUBERNETES_PUBLIC_ADDRESS \
--ports 6443 \
--region (gcloud config get-value compute/region) \
--target-pool kubernetes-target-pool
Created [https://www.googleapis.com/compute/v1/projects/sandbox-project/regions/asia-northeast1/forwardingRules/kubernetes-forwarding-rule].
每个实例的准备工作
我们将按照《公式文档的Installing kubeadm》进行参考和操作。
确认目标实例
$ gcloud compute instances list --filter="tags.items=kubernetes-by-kubeadm"
NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
controller asia-northeast1-a n1-standard-4 10.240.0.10 35.221.119.152 RUNNING
worker-0 asia-northeast1-a n1-standard-4 10.240.0.20 35.221.99.135 RUNNING
worker-1 asia-northeast1-a n1-standard-4 10.240.0.21 34.84.119.161 RUNNING
worker-2 asia-northeast1-a n1-standard-4 10.240.0.22 34.85.61.122 RUNNING
请连接到每个实例并执行 gcloud compute ssh $INSTANCE_NAME 命令,然后进行各种准备工作。
再次提醒,实例内的shell是bash。
使iptables能够处理通过桥接的流量
hitsumabushi845@controller:~$ sudo modprobe br_netfilter
hitsumabushi845@controller:~$ lsmod | grep br_netfilter
br_netfilter 28672 0
bridge 176128 1 br_netfilter
hitsumabushi845@controller:~$ cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
> net.bridge.bridge-nf-call-ip6tables = 1
> net.bridge.bridge-nf-call-iptables = 1
> EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
hitsumabushi845@controller:~$ sudo sysctl --system
* Applying /etc/sysctl.d/10-console-messages.conf ...
kernel.printk = 4 4 1 7
* Applying /etc/sysctl.d/10-ipv6-privacy.conf ...
net.ipv6.conf.all.use_tempaddr = 2
net.ipv6.conf.default.use_tempaddr = 2
* Applying /etc/sysctl.d/10-kernel-hardening.conf ...
kernel.kptr_restrict = 1
* Applying /etc/sysctl.d/10-link-restrictions.conf ...
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
* Applying /etc/sysctl.d/10-magic-sysrq.conf ...
kernel.sysrq = 176
* Applying /etc/sysctl.d/10-network-security.conf ...
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.all.rp_filter = 2
* Applying /etc/sysctl.d/10-ptrace.conf ...
kernel.yama.ptrace_scope = 1
* Applying /etc/sysctl.d/10-zeropage.conf ...
vm.mmap_min_addr = 65536
* Applying /usr/lib/sysctl.d/50-default.conf ...
net.ipv4.conf.default.promote_secondaries = 1
sysctl: setting key "net.ipv4.conf.all.promote_secondaries": Invalid argument
net.ipv4.ping_group_range = 0 2147483647
net.core.default_qdisc = fq_codel
fs.protected_regular = 1
fs.protected_fifos = 1
* Applying /usr/lib/sysctl.d/50-pid-max.conf ...
kernel.pid_max = 4194304
* Applying /etc/sysctl.d/60-gce-network-security.conf ...
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 1
net.ipv4.conf.default.secure_redirects = 1
net.ipv4.ip_forward = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.log_martians = 1
kernel.randomize_va_space = 2
kernel.panic = 10
* Applying /etc/sysctl.d/99-cloudimg-ipv6.conf ...
net.ipv6.conf.all.use_tempaddr = 0
net.ipv6.conf.default.use_tempaddr = 0
* Applying /etc/sysctl.d/99-sysctl.conf ...
* Applying /etc/sysctl.d/k8s.conf ...
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
* Applying /usr/lib/sysctl.d/protect-links.conf ...
fs.protected_fifos = 1
fs.protected_hardlinks = 1
fs.protected_regular = 2
fs.protected_symlinks = 1
* Applying /etc/sysctl.conf ...
安装 containerd
我们将按照Container运行时中containerd相关部分的内容来进行。
hitsumabushi845@controller:~$ cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
> overlay
> br_netfilter
> EOF
overlay
br_netfilter
hitsumabushi845@controller:~$ sudo modprobe overlay
hitsumabushi845@controller:~$ sudo modprobe br_netfilter
hitsumabushi845@controller:~$ cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
> net.bridge.bridge-nf-call-iptables = 1
> net.ipv4.ip_forward = 1
> net.bridge.bridge-nf-call-ip6tables = 1
> EOF
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
hitsumabushi845@controller:~$ sudo sysctl --system
(略)
hitsumabushi845@controller:~$ sudo apt-get update && sudo apt-get install -y containerd
(略)
hitsumabushi845@controller:~$ sudo mkdir -p /etc/containerd
hitsumabushi845@controller:~$ containerd config default | sudo tee /etc/containerd/config.toml
(略)
hitsumabushi845@controller:~$ sudo systemctl restart containerd
hitsumabushi845@controller:~$ sudo systemctl status containerd
● containerd.service - containerd container runtime
Loaded: loaded (/lib/systemd/system/containerd.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2021-02-21 10:51:28 UTC; 10s ago
Docs: https://containerd.io
Process: 14555 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
Main PID: 14562 (containerd)
Tasks: 14
Memory: 21.3M
CGroup: /system.slice/containerd.service
└─14562 /usr/bin/containerd
Feb 21 10:51:28 controller containerd[14562]: time="2021-02-21T10:51:28.302856785Z" level=error msg="Failed to load cni during init, please check CRI plugin status before setting up network for pods" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
Feb 21 10:51:28 controller containerd[14562]: time="2021-02-21T10:51:28.303184344Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
Feb 21 10:51:28 controller containerd[14562]: time="2021-02-21T10:51:28.303712626Z" level=info msg="Start subscribing containerd event"
Feb 21 10:51:28 controller containerd[14562]: time="2021-02-21T10:51:28.303792746Z" level=info msg="Start recovering state"
Feb 21 10:51:28 controller containerd[14562]: time="2021-02-21T10:51:28.303902734Z" level=info msg=serving... address=/run/containerd/containerd.sock.ttrpc
Feb 21 10:51:28 controller containerd[14562]: time="2021-02-21T10:51:28.303999145Z" level=info msg="Start event monitor"
Feb 21 10:51:28 controller containerd[14562]: time="2021-02-21T10:51:28.304029295Z" level=info msg=serving... address=/run/containerd/containerd.sock
Feb 21 10:51:28 controller containerd[14562]: time="2021-02-21T10:51:28.304054635Z" level=info msg="containerd successfully booted in 0.042356s"
Feb 21 10:51:28 controller containerd[14562]: time="2021-02-21T10:51:28.304030429Z" level=info msg="Start snapshots syncer"
Feb 21 10:51:28 controller containerd[14562]: time="2021-02-21T10:51:28.304557561Z" level=info msg="Start streaming server"
kubeadm、kubelet、kubectl 的安装
返回到公式文档的“安装 kubeadm”部分。
hitsumabushi845@controller:~$ sudo apt-get update && sudo apt-get install -y apt-transport-https curl
(略)
hitsumabushi845@controller:~$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
OK
hitsumabushi845@controller:~$ cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
> deb https://apt.kubernetes.io/ kubernetes-xenial main
> EOF
deb https://apt.kubernetes.io/ kubernetes-xenial main
hitsumabushi845@controller:~$ sudo apt-get update
(略)
hitsumabushi845@controller:~$ sudo apt-get install -y kubelet kubeadm kubectl
(略)
hitsumabushi845@controller:~$ sudo apt-mark hold kubelet kubeadm kubectl
kubelet set on hold.
kubeadm set on hold.
kubectl set on hold.
这些任务将在控制器、工作机-0、工作机-1和工作机-2的各个实例上执行。
创建控制平面
从这里开始,按照使用kubeadm创建集群的步骤进行。
执行 kubeadm init。
这次我们将分配外部IP作为API服务器的终结点,因此在–control-plane-endpoint中指定外部IP。此外,Pod CIDR被设置为10.200.0.0/16。
hitsumabushi845@controller:~$ sudo kubeadm init --control-plane-endpoint=34.85.15.20 --pod-network-cidr=10.200.0.0/16
[init] Using Kubernetes version: v1.20.4
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [controller kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.240.0.10 34.85.15.20]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [controller localhost] and IPs [10.240.0.10 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [controller localhost] and IPs [10.240.0.10 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[apiclient] All control plane components are healthy after 104.006101 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.20" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node controller as control-plane by adding the labels "node-role.kubernetes.io/master=''" and "node-role.kubernetes.io/control-plane='' (deprecated)"
[mark-control-plane] Marking the node controller as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 3cznxo.v1ax148y0hjdzail
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:
kubeadm join 34.85.15.20:6443 --token 3cznxo.v1ax148y0hjdzail \
--discovery-token-ca-cert-hash sha256:d778f85f07c092a196b77e1669dfceed74b9092587293274fcc8652a9936511f \
--control-plane
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 34.85.15.20:6443 --token 3cznxo.v1ax148y0hjdzail \
--discovery-token-ca-cert-hash sha256:d778f85f07c092a196b77e1669dfceed74b9092587293274fcc8652a9936511f
当成功初始化后,在底部会显示出各种命令。
如果不是以超级用户身份使用kubectl命令,则在下方敲击。
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
执行。
如果要添加控制平面节点,可以这样做
kubeadm join 34.85.15.20:6443 --token 3cznxo.v1ax148y0hjdzail \
--discovery-token-ca-cert-hash sha256:d778f85f07c092a196b77e1669dfceed74b9092587293274fcc8652a9936511f \
--control-plane
执行。
如果要添加工作节点,
kubeadm join 34.85.15.20:6443 --token 3cznxo.v1ax148y0hjdzail \
--discovery-token-ca-cert-hash sha256:d778f85f07c092a196b77e1669dfceed74b9092587293274fcc8652a9936511f
保留执行特定命令,尤其是在后续工作中需要添加工作节点的命令。
请确认 kubectl 的执行情况。
使用先前提到的命令,确保控制器实例可以以普通用户身份执行 kubectl。
hitsumabushi845@controller:~$ mkdir -p $HOME/.kube
hitsumabushi845@controller:~$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
hitsumabushi845@controller:~$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
hitsumabushi845@controller:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:12:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:03:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
确认是否通过使用 “kubectl get nodes” 命令获得返回结果。
hitsumabushi845@controller:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
controller NotReady control-plane,master 47m v1.20.4
控制器显示为节点,但由于被标记为 NoSchedule 的 Taint,通常不会将 Pod 调度到这个节点。
hitsumabushi845@controller:~$ kubectl describe node controller | grep Taint
Taints: node-role.kubernetes.io/master:NoSchedule
安装Calico。
下载 calico.yaml 文件。
hitsumabushi845@controller:~$ sudo curl -OL https://docs.projectcalico.org/manifests/calico.yaml
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 20847 100 20847 0 0 50112 0 --:--:-- --:--:-- --:--:-- 50233
由于在执行 kubeadm init 时指定了 –pod-network-cidr,所以需要修改 calico.yaml 中 CALICO_IPV4POOL_CIDR 的值。
另外,为了保险起见,明确将 FELIX_IPTABLESBACKEND 的值设置为 NFT(默认为 Auto)。
- # - name: CALICO_IPV4POOL_CIDR
- # value: "192.168.0.0/16"
+ - name: CALICO_IPV4POOL_CIDR
+ value: "10.200.0.0/16"
+ - name: FELIX_IPTABLESBACKEND
+ value: NFT
hitsumabushi845@controller:~$ kubectl apply -f calico.yaml
configmap/calico-config created
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
hitsumabushi845@controller:~$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-6b8f6f78dc-k84wt 1/1 Running 0 5m
kube-system calico-node-4jrkc 1/1 Running 0 5m
kube-system coredns-74ff55c5b-5ctxr 1/1 Running 0 52m
kube-system coredns-74ff55c5b-95lxc 1/1 Running 0 52m
kube-system etcd-controller 1/1 Running 0 52m
kube-system kube-apiserver-controller 1/1 Running 1 52m
kube-system kube-controller-manager-controller 1/1 Running 0 52m
kube-system kube-proxy-2sgv7 1/1 Running 0 52m
kube-system kube-scheduler-controller 1/1 Running 0 52m
hitsumabushi845@controller:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
controller Ready control-plane,master 53m v1.20.4
创建Worker节点
由于目前为止的工作已经完成了控制平面的创建,所以接下来我们将添加工作节点。
执行 kubeadm join
在工作节点的实例上执行上述的kubeadm join命令。
hitsumabushi845@worker-0:~$ sudo kubeadm join 34.85.15.20:6443 --token 3cznxo.v1ax148y0hjdzail --discovery-token-ca-cert-hash sha256:d778f85f07c092a196b77e1669dfceed74b9092587293274fcc8652a9936511f
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
确定
在控制平面实例上执行 kubectl get nodes 命令。
hitsumabushi845@controller:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
controller Ready control-plane,master 57m v1.20.4
worker-0 Ready <none> 73s v1.20.4
我已经确认添加了工作节点。
同样地,我们还需要在worker-1和worker-2的实例上进行操作,以确保有3个工作节点存在。
hitsumabushi845@controller:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
controller Ready control-plane,master 72m v1.20.4
worker-0 Ready <none> 16m v1.20.4
worker-1 Ready <none> 4m49s v1.20.4
worker-2 Ready <none> 26s v1.20.4
让本地能够使用kubectl命令行工具
从控制平面实例中复制kubeconfig文件。
$ gcloud compute scp root@controller:/etc/kubernetes/admin.conf .
admin.conf
确认使用 –kubeconfig 选项指定 admin.conf 的副本后,kubectl 能够执行命令。
如果每次指定 –kubeconfig 很麻烦,可以设置 $KUBECONFIG 环境变量来简化操作。
$ k get nodes --kubeconfig ./admin.conf
NAME STATUS ROLES AGE VERSION
controller Ready control-plane,master 74m v1.20.4
worker-0 Ready <none> 18m v1.20.4
worker-1 Ready <none> 6m56s v1.20.4
worker-2 Ready <none> 2m33s v1.20.4
确认操作
在这个阶段,我们已经成功创建了一个拥有1个控制平面节点和3个工作节点的集群。我们将参考kubernetes-the-hard-way中的Smoke Test来验证集群的运行情况。
创建部署
确认能够创建部署资源。
$ k create deploy nginx --image=nginx --replicas=3 --kubeconfig ./admin.conf
deployment.apps/nginx created
$ k get deploy --kubeconfig ./admin.conf
NAME READY UP-TO-DATE AVAILABLE AGE
nginx 3/3 3 3 47s
$ k get po -owide --kubeconfig ./admin.conf
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-6799fc88d8-69bjt 1/1 Running 0 29m 10.200.43.2 worker-0 <none> <none>
nginx-6799fc88d8-8gdqj 1/1 Running 0 29m 10.200.133.194 worker-2 <none> <none>
nginx-6799fc88d8-d92bc 1/1 Running 0 29m 10.200.226.66 worker-1 <none> <none>
确认端口转发
我要尝试对已创建的nginx Pod进行端口转发。
$ set POD_NAME (k get po -l app=nginx -o jsonpath="{.items[0].metadata.name}" --kubeconfig ./admin.conf)
$ echo $POD_NAME
nginx-6799fc88d8-4ttf9
$ k port-forward $POD_NAME 8080:80 --kubeconfig ./admin.conf
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
从另一个终端窗口中,确认可以对进行端口转发的 Pod 使用 curl 命令。
$ curl --head http://127.0.0.1:8080
HTTP/1.1 200 OK
Server: nginx/1.19.7
Date: Sun, 21 Feb 2021 13:46:42 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 16 Feb 2021 15:57:18 GMT
Connection: keep-alive
ETag: "602beb5e-264"
Accept-Ranges: bytes
为了确认,我会查看目标 Pod 的日志。
在日志的最底部可以看到刚才的 curl 请求的日志。
$ k logs $POD_NAME --kubeconfig ./admin.conf
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
127.0.0.1 - - [21/Feb/2021:13:46:42 +0000] "HEAD / HTTP/1.1" 200 0 "-" "curl/7.64.1" "-"
使用NodePort服务将部署公开到外部
我們將使用NodePort服務來測試先前創建的部署的外部公開。
$ k expose deploy nginx --port 80 --type NodePort --kubeconfig ./admin.conf
service/nginx exposed
$ set NODE_PORT (k get svc nginx --output=jsonpath='{range .spec.ports[0]}{.nodePort}' --kubeconfig ./admin.conf)
$ echo $NODE_PORT
31120
在第31120号端口上部署已经发布,但是由于VPC的防火墙规则尚未允许对该端口的外部通信,因此需要添加规则。
$ gcloud compute firewall-rules create kubernetes-by-kubeadm-allow-nginx-service \
--allow=tcp:{$NODE_PORT} \
--network kubernetes-by-kubeadm
Creating firewall...⠹Created [https://www.googleapis.com/compute/v1/projects/sandbox-project/global/firewalls/kubernetes-by-kubeadm-allow-nginx-service].
Creating firewall...done.
NAME NETWORK DIRECTION PRIORITY ALLOW DENY DISABLED
kubernetes-by-kubeadm-allow-nginx-service kubernetes-by-kubeadm INGRESS 1000 tcp:31120 False
我們新增了一條規則,以確保與31120號端口的外部連通性,接下來要確認curl是否可使用。
$ set EXTERNAL_IP (gcloud compute instances describe worker-0 --format 'value(networkInterfaces[0].accessConfigs[0].natIP)')
$ curl -I http://{$EXTERNAL_IP}:{$NODE_PORT}
HTTP/1.1 200 OK
Server: nginx/1.19.7
Date: Sun, 21 Feb 2021 13:54:45 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 16 Feb 2021 15:57:18 GMT
Connection: keep-alive
ETag: "602beb5e-264"
Accept-Ranges: bytes
我确认通过使用NodePort服务可以将Deployment外部公开。
整理收拾
因此,我们能在GCE上构建了一个k8s集群。
最后我们将进行清理工作,删除GCP上的各种资源。
删除 GCE 实例
$ gcloud -q compute instances delete \
controller worker-0 worker-1 worker-2 \
--zone (gcloud config get-value compute/zone)
Deleted [https://www.googleapis.com/compute/v1/projects/sandbox-project/zones/asia-northeast1-a/instances/controller].
Deleted [https://www.googleapis.com/compute/v1/projects/sandbox-project/zones/asia-northeast1-a/instances/worker-0].
Deleted [https://www.googleapis.com/compute/v1/projects/sandbox-project/zones/asia-northeast1-a/instances/worker-1].
Deleted [https://www.googleapis.com/compute/v1/projects/sandbox-project/zones/asia-northeast1-a/instances/worker-2].
删除网络资源
删除外部负载均衡器
$ gcloud -q compute forwarding-rules list
NAME REGION IP_ADDRESS IP_PROTOCOL TARGET
kubernetes-forwarding-rule asia-northeast1 34.85.15.20 TCP asia-northeast1/targetPools/kubernetes-target-pool
$ gcloud -q compute forwarding-rules delete kubernetes-forwarding-rule \
--region (gcloud config get-value compute/region)
Deleted [https://www.googleapis.com/compute/v1/projects/sandbox-project/regions/asia-northeast1/forwardingRules/kubernetes-forwarding-rule].
$ gcloud -q compute forwarding-rules list
Listed 0 items.
$ gcloud -q compute target-pools list
NAME REGION SESSION_AFFINITY BACKUP HEALTH_CHECKS
kubernetes-target-pool asia-northeast1 NONE kubernetes
$ gcloud -q compute target-pools delete kubernetes-target-pool
Deleted [https://www.googleapis.com/compute/v1/projects/sandbox-project/regions/asia-northeast1/targetPools/kubernetes-target-pool].
$ gcloud -q compute target-pools list
Listed 0 items.
$ gcloud -q compute http-health-checks list
NAME HOST PORT REQUEST_PATH
kubernetes kubernetes.default.svc.cluster.local 80 /healthz
$ gcloud -q compute http-health-checks delete kubernetes
Deleted [https://www.googleapis.com/compute/v1/projects/sandbox-project/global/httpHealthChecks/kubernetes].
$ gcloud -q compute http-health-checks list
Listed 0 items.
$ gcloud -q compute addresses list
NAME ADDRESS/RANGE TYPE PURPOSE NETWORK REGION SUBNET STATUS
kubernetes-by-kubeadm 34.85.15.20 EXTERNAL asia-northeast1 RESERVED
$ gcloud -q compute addresses delete kubernetes-by-kubeadm
Deleted [https://www.googleapis.com/compute/v1/projects/sandbox-project/regions/asia-northeast1/addresses/kubernetes-by-kubeadm].
$ gcloud -q compute addresses list
Listed 0 items.
删除防火墙规则
$ gcloud -q compute firewall-rules list
NAME NETWORK DIRECTION PRIORITY ALLOW DENY DISABLED
default-allow-icmp default INGRESS 65534 icmp False
default-allow-internal default INGRESS 65534 tcp:0-65535,udp:0-65535,icmp False
default-allow-rdp default INGRESS 65534 tcp:3389 False
default-allow-ssh default INGRESS 65534 tcp:22 False
kubernetes-by-kubeadm-allow-external kubernetes-by-kubeadm INGRESS 1000 tcp:22,tcp:6443,icmp False
kubernetes-by-kubeadm-allow-health-check kubernetes-by-kubeadm INGRESS 1000 tcp False
kubernetes-by-kubeadm-allow-internal kubernetes-by-kubeadm INGRESS 1000 tcp,udp,icmp False
kubernetes-by-kubeadm-allow-nginx-service kubernetes-by-kubeadm INGRESS 1000 tcp:31120 False
To show all fields of the firewall, please show in JSON format: --format=json
To show all fields in table format, please see the examples in --help.
$ gcloud -q compute firewall-rules delete \
kubernetes-by-kubeadm-allow-external \
kubernetes-by-kubeadm-allow-internal \
kubernetes-by-kubeadm-allow-health-check \
kubernetes-by-kubeadm-allow-nginx-service
Deleted [https://www.googleapis.com/compute/v1/projects/sandbox-project/global/firewalls/kubernetes-by-kubeadm-allow-external].
Deleted [https://www.googleapis.com/compute/v1/projects/sandbox-project/global/firewalls/kubernetes-by-kubeadm-allow-internal].
Deleted [https://www.googleapis.com/compute/v1/projects/sandbox-project/global/firewalls/kubernetes-by-kubeadm-allow-health-check].
Deleted [https://www.googleapis.com/compute/v1/projects/sandbox-project/global/firewalls/kubernetes-by-kubeadm-allow-nginx-service].
$ gcloud -q compute firewall-rules list
NAME NETWORK DIRECTION PRIORITY ALLOW DENY DISABLED
default-allow-icmp default INGRESS 65534 icmp False
default-allow-internal default INGRESS 65534 tcp:0-65535,udp:0-65535,icmp False
default-allow-rdp default INGRESS 65534 tcp:3389 False
default-allow-ssh default INGRESS 65534 tcp:22 False
To show all fields of the firewall, please show in JSON format: --format=json
To show all fields in table format, please see the examples in --help.
删除VPC子网
$ gcloud -q compute networks subnets list --filter="network:kubernetes-by-kubeadm"
NAME REGION NETWORK RANGE
kubernetes asia-northeast1 kubernetes-by-kubeadm 10.240.0.0/24
$ gcloud -q compute networks subnets delete kubernetes
Deleted [https://www.googleapis.com/compute/v1/projects/sandbox-project/regions/asia-northeast1/subnetworks/kubernetes].
$ gcloud -q compute networks subnets list --filter="network:kubernetes-by-kubeadm"
Listed 0 items.
删除 VPC 网络
$ gcloud -q compute networks list
NAME SUBNET_MODE BGP_ROUTING_MODE IPV4_RANGE GATEWAY_IPV4
default AUTO REGIONAL
kubernetes-by-kubeadm CUSTOM REGIONAL
$ gcloud -q compute networks delete kubernetes-by-kubeadm
Deleted [https://www.googleapis.com/compute/v1/projects/sandbox-project/global/networks/kubernetes-by-kubeadm].
$ gcloud -q compute networks list
NAME SUBNET_MODE BGP_ROUTING_MODE IPV4_RANGE GATEWAY_IPV4
default AUTO REGIONAL