降低树莓派上 Kubernetes 和 Docker 的版本的讨论
首先
※２０２０年の年末の話です。将来のバージョンでは普通に動くんだと思います。
※あと、超初心者なので何か忘れているとか、余計なことやっているとか
その他諸々あるのかもしれません。ゴメンナサイ。
※もっとこうすればいいのにとかこうやれば解決できますよ〜とかありましたら、
こっそり教えてください^^
※作ったときの話はまた今度〜
首先，在安装时
Docker编译
立即安装！
$ curl -sSL https://get.docker.com/ | sh
# Executing docker install script, commit: 3d8fe77c2c46c5b7571f94b42793905e5b3e42e4
ーーー中略ーーー
+ sudo -E sh -c docker version
Client: Docker Engine - Community
Version: 20.10.1
API version: 1.41
Go version: go1.13.15
Git commit: 831ebea
Built: Tue Dec 15 04:35:27 2020
OS/Arch: linux/arm
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.1
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: f001486
Built: Tue Dec 15 04:33:02 2020
OS/Arch: linux/arm
Experimental: false
containerd:
Version: 1.4.3
GitCommit: 269548fa27e0089a8b8278fc4fc781d7f65a939b
runc:
Version: 1.0.0-rc92
GitCommit: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
docker-init:
Version: 0.19.0
GitCommit: de40ad0
ーーー以下略ーーー
というわけでDockerのバージョンは20.10.1
あとから思えばここから、ずれていたんだなぁ。
今(2020年の年末)のKubernetesの対応外だけど何とかなるでしょと、、、
Workerになるラズパイにも同じようにインストール！
1-2. Kubernetes編
既安装了Docker，接下来是Kubernetes！完成Docker的usermod、apt-mark以及导入google的apt-key等操作。
$ sudo apt-get install kubelet kubeadm kubectl kubernetes-cni
ーーー中略ーーー
Unpacking kubeadm (1.20.1-00) ...
Setting up conntrack (1:1.4.5-2) ...
Setting up kubectl (1.20.1-00) ...
Setting up ebtables (2.0.10.4+snapshot20181205-3) ...
Setting up socat (1.7.3.2-2) ...
Setting up cri-tools (1.13.0-01) ...
Setting up kubernetes-cni (0.8.7-00) ...
Setting up kubelet (1.20.1-00) ...```
Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service → /lib/systemd/system/kubelet.service.
Setting up kubeadm (1.20.1-00) ...
Processing triggers for man-db (2.8.5-2) ...
Processing triggers for libc-bin (2.28-10+rpi1) ...
まぁ、これは普通にインストール終了。
というわけで、Kubernetesは1.20.1。
Workerになるラズパイにも同じようにインストール
从这里开始，由初学者的战斗开始了~ 哈哈
2. 无法执行kubeadm init。
本次使用的树莓派是Raspberry Pi 4 Model B / 8GB，所以即使在图形用户界面的操作系统中，内存也是这样的。
$ free -h
total used free shared buff/cache available
Mem: 7.7Gi 617Mi 5.8Gi 574Mi 1.3Gi 6.3Gi
Swap: 0B 0B 0B
所以我們立即在成為主人的樹莓派上進行了kubeadm init。
$ sudo kubeadm init --pod-network-cidr=10.244.0.0/16
[init] Using Kubernetes version: v1.20.1
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.1. Latest validated version: 19.03
[WARNING SystemVerification]: missing optional cgroups: hugetlb
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR Mem]: the system RAM (1 MB) is less than the minimum 1700 MB
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
なんじゃこりゃ。。。
Dockerのバージョンが高いぜ！は良いとして(良くない)
このシステム、メモリが1MBしかないんですが、最低でも1700MB要ります〜！
ってどういうこと？？？
さっきの空きメモリはどこ行ってしまったんだ。。。
というわけでいろいろ漁ってみたんですがよくわからないので、
オプションに
–ignore-preflight-errors=Mem
を指定して、エラー無視して逃げちゃおうｗ
$ sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=Mem
[init] Using Kubernetes version: v1.20.1
[preflight] Running pre-flight checks
[WARNING Mem]: the system RAM (1 MB) is less than the minimum 1700 MB
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.1. Latest validated version: 19.03
[WARNING SystemVerification]: missing optional cgroups: hugetlb
[preflight] Pulling images required for setting up a Kubernetes cluster
ーーー中略ーーー
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
ーーー以下略ーーー
这次安装好像成功了（开个玩笑，其实并没有成功）
完成了各种环境配置，从每个Worker树莓派上运行kubeadm join!!!
如果能加入，立即查看节点!
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-m Ready control-plane,master 5m23s v1.20.1
k8s-w1 Ready <none> 3m56s v1.20.1
k8s-w2 Ready <none> 3m41s v1.20.1
k8s-w3 Ready <none> 3m20s v1.20.1
嘿，原来它可以正常地运转啊~（在那时候，我这样想着…）
3. 总感觉有点奇怪
因为可以从主节点(master)查看所有节点(node)，所以我安装了k8s，但发现似乎不足为惧。
我一边得意地笑着吃饭一边回来，也许再次看看节点吧~
$ kubectl get node
The connection to the server <IP>:6443 was refused - did you specify the right host or port?
$ kubectl get node
The connection to the server <IP>:6443 was refused - did you specify the right host or port?
...
ーーー２,3分この状態ーーー
$ kubectl get node
Unable to connect to the server: net/http: TLS handshake timeout
$ kubectl get node
Unable to connect to the server: net/http: TLS handshake timeout
...
ーーー数１０秒この状態ーーー
$ kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-m Ready control-plane,master 1h v1.20.1
k8s-w1 Ready worker 1h v1.20.1
k8s-w2 Ready worker 1h v1.20.1
k8s-w3 Ready worker 1h v1.20.1
ーーー復活ーーー
这到底是怎么回事？
查看了日志，发现了致命错误，重启了也能理解，但还是无能为力…
这种状态每隔十分钟就会重复一次，真是让人受罪…
纠结了很久，最后决定降低版本来解决。
4. 卸载 Kubernetes
师傅，工作人员应该在所有节点上进行操作。
虽然实际上最好停止服务，但唉，天气湿热呢ｗ
删除Kubernetes的设置。
如果不做这个，重新安装后将会引用旧的设置。-我
$ sudo kubeadm reset
4-2. 解除握持
首先检查处于保持状态的内容，将相关项解除保持，以防止版本在 apt update 等操作中自动升级。
$ sudo apt-mark showhold
$ sudo apt-mark unhold kubelet kubeadm kubectl kubernetes-cni
4-2. 卸载Kubernetes
使用purge命令删除并用autoremove命令清除相关内容。
$ sudo apt-get purge -y kubeadm kubectl kubelet kubernetes-cni kube*
$ sudo apt-get autoremove -y
4-3. 树莓派重新启动
因为感觉有点不舒服，所以我重新启动了所有的树莓派。
（其实我在这里安装了 Kubernetes 1.19.6，但是上面的获取节点的连接被拒问题没有改善，呵呵）
5. 卸载Docker
这件事情我们将在全台推行。
解除抓住
首先，请确认当前处于 hold 状态的物品，并确保 Docker 也不会因为 apt update 等操作而自动升级。
$ sudo apt-mark showhold
$ sudo apt-mark unhold docker-ce docker-ce-cli
5-2. 卸载Docker
purgeして関連削除
$ sudo apt-get purge -y docker-ce docker-ce-cli
$ sudo apt-get autoremove -y
5-3. 树莓派重新启动
因为感觉有点不舒服，所以我让所有的树莓派重新启动哈哈。
重新安装
Dockerを19.03系、
Kubernetesを1.19系で再インストールしてみる
6-1. Docker (原生语言中文翻译选项)：
选择版本
首先，尝试查找最新的19.03版本是什么。
$ apt-cache madison docker-ce
docker-ce | 5:20.10.1~3-0~raspbian-buster | https://download.docker.com/linux/raspbian buster/stable armhf Packages
docker-ce | 5:20.10.0~3-0~raspbian-buster | https://download.docker.com/linux/raspbian buster/stable armhf Packages
docker-ce | 5:19.03.14~3-0~raspbian-buster | https://download.docker.com/linux/raspbian buster/stable armhf Packages
docker-ce | 5:19.03.13~3-0~raspbian-buster | https://download.docker.com/linux/raspbian buster/stable armhf Packages
docker-ce | 5:19.03.12~3-0~raspbian-buster | https://download.docker.com/linux/raspbian buster/stable armhf Packages
ーーー中略ーーー
docker-ce | 5:19.03.2~3-0~raspbian-buster | https://download.docker.com/linux/raspbian buster/stable armhf Packages
docker-ce | 5:19.03.1~3-0~raspbian-buster | https://download.docker.com/linux/raspbian buster/stable armhf Packages
19.03版本看起来不错。
我还查了一下docker-ce-cli，也是同样的19.03.14版本。
所以，决定使用Docker 19.03.14！
6-1-2. 安装时指定版本。
$ sudo apt-get install -y --no-install-recommends docker-ce=5:19.03.14~3-0~raspbian-buster docker-ce-cli=5:19.03.14~3-0~raspbian-buster
6-1-3. 確認
$ docker version
Client: Docker Engine - Community
Version: 19.03.14
API version: 1.40
Go version: go1.13.15
Git commit: 5eb3275d40
Built: Tue Dec 1 19:21:06 2020
OS/Arch: linux/arm
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.14
API version: 1.40 (minimum version 1.12)
Go version: go1.13.15
Git commit: 5eb3275d40
Built: Tue Dec 1 19:19:00 2020
OS/Arch: linux/arm
Experimental: false
containerd:
Version: 1.4.3
GitCommit: 269548fa27e0089a8b8278fc4fc781d7f65a939b
runc:
Version: 1.0.0-rc92
GitCommit: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
docker-init:
Version: 0.18.0
GitCommit: fec3683
6-2. Kubernetes 六到二的 Kubernetes
6-2-1. バージョン選定
こちらもバージョン確認
$ sudo apt-cache madison kubeadm
kubeadm | 1.20.1-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
kubeadm | 1.20.0-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
kubeadm | 1.19.6-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
kubeadm | 1.19.5-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
kubeadm | 1.19.4-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
kubeadm | 1.19.3-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
ーーー中略ーーー
kubeadm | 1.6.0-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
kubeadm | 1.5.7-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
kubeadm | 1.5.6-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
$
$
$ sudo apt-cache madison kubelet
kubelet | 1.20.1-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
kubelet | 1.20.0-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
kubelet | 1.19.6-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
kubelet | 1.19.5-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
kubelet | 1.19.4-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
kubelet | 1.19.3-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
ーーー中略ーーー
kubelet | 1.5.2-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
kubelet | 1.5.1-01 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
kubelet | 1.5.1-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
$
$
$ sudo apt-cache madison kubectl
kubectl | 1.20.1-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
kubectl | 1.20.0-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
kubectl | 1.19.6-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
kubectl | 1.19.5-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
kubectl | 1.19.4-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
kubectl | 1.19.3-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
ーーー中略ーーー
kubectl | 1.5.3-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
kubectl | 1.5.2-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
kubectl | 1.5.1-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
1. 看起来1.19版本不错。
所以，我们决定使用Kubernetes的1.19.6版本！
至于kubernetes-cni…
$ sudo apt-cache madison kubernetes-cni
kubernetes-cni | 0.8.7-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
kubernetes-cni | 0.8.6-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
kubernetes-cni | 0.7.5-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
kubernetes-cni | 0.6.0-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
kubernetes-cni | 0.5.1-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
kubernetes-cni | 0.3.0.1-07a8a2-00 | http://apt.kubernetes.io kubernetes-xenial/main armhf Packages
无所谓，最新的也可以（笑）
6-2-2. 选择指定版本进行安装。
sudo apt-get install -y --no-install-recommends kubelet=1.19.6-00 kubeadm=1.19.6-00 kubectl=1.19.6-00 kubernetes-cni
6-2-3. 版本确认
$ kubelet --version
Kubernetes v1.19.6
$
$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.6", GitCommit:"fbf646b339dc52336b55d8ec85c181981b86331a", GitTreeState:"clean", BuildDate:"2020-12-18T12:07:25Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/arm"}
$
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.6", GitCommit:"fbf646b339dc52336b55d8ec85c181981b86331a", GitTreeState:"clean", BuildDate:"2020-12-18T12:09:30Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/arm"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.6", GitCommit:"fbf646b339dc52336b55d8ec85c181981b86331a", GitTreeState:"clean", BuildDate:"2020-12-18T12:01:36Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/arm"}
6-3. 所以
一切平安无事
Docker版本为19.03.14
Kubernetes版本为1.19.6
＼(^O^)／
所以，尽管kubeadm init出现了一些警告，但没有发生1MB内存问题，并且成功安装了！
$ sudo kubeadm init --pod-network-cidr=10.244.0.0/16
I0102 01:10:31.569593 5226 version.go:252] remote version is much newer: v1.20.1; falling back to: stable-1.19
W0102 01:10:31.967024 5226 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.19.6
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: missing optional cgroups: hugetlb
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
通过调整节点，不再出现拒绝连接的问题了ー＼(^O^)／
$ kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-m Ready master 10h11m v1.19.6
k8s-w1 Ready <none> 10h15m v1.19.6
k8s-w2 Ready <none> 10h16m v1.19.6
k8s-w3 Ready <none> 10h18m v1.19.6
结束了。
