今回は(今回も)CentOS 7.7 1908 Minimalを素で入れて、公式ツールであるkubeadmを使ってKubernetesクラスタを作ってみた。
何だかんだで一度もやったことがなかった基礎…
kubernetes.io
kubernetes.io
追記2: masterを複数ノードにする場合は、kubeadm initに追加のオプションが必要っぽい
zaki-hmkc.hatenablog.com
追記1: ちなみにクラスタはデプロイできたけど、firewalldが有効だとアプリケーションpodの通信がうまくいってないです。
OSのインストール
普通に。
ちなみにH/W要件は「2core」「RAM2GB(2GBの場合アプリの余裕無し)」とあるので、構成は以下の通り
| node |
CPU |
RAM |
| master |
4 |
8GB |
| worker |
4 |
8GB |
とりあえず1台ずつのみ。
また、ネットワーク内のDNS(dnsmasq使用)で、名前解決できるようになってる。
# k8s
192.168.0.121 k8s-master01.esxi.jp-z.jp
192.168.0.125 k8s-node01.esxi.jp-z.jp
192.168.0.121 k8s-master.esxi.jp-z.jp # LB
192.168.0.121 api.k8s.esxi.jp-z.jp # LB
マルチノードに備えて、LB用のFQDNも設定してる(今のところ出番なし)
OS/ミドルウェア設定(master/worker共通)
インストール後の状態
[zaki@k8s-master01 ~]$ cat /etc/redhat-release
CentOS Linux release 7.7.1908 (Core)
[zaki@k8s-master01 ~]$ df -h
ファイルシス サイズ 使用 残り 使用% マウント位置
devtmpfs 3.8G 0 3.8G 0% /dev
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 3.9G 12M 3.8G 1% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/mapper/centos-root 56G 1.2G 55G 3% /
/dev/sda1 1014M 150M 865M 15% /boot
tmpfs 781M 0 781M 0% /run/user/1000
[zaki@k8s-master01 ~]$ free -h
total used free shared buff/cache available
Mem: 7.6G 237M 7.2G 11M 203M 7.2G
Swap: 3.5G 0B 3.5G
swap off
[zaki@k8s-master01 ~]$ sudo swapoff -a
[zaki@k8s-master01 ~]$ free -h
total used free shared buff/cache available
Mem: 7.6G 235M 7.2G 11M 204M 7.2G
Swap: 0B 0B 0B
[zaki@k8s-master01 ~]$ cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Thu Mar 19 06:59:37 2020
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root / xfs defaults 0 0
UUID=6a6cc094-d014-4870-b45a-96a3d2a858e4 /boot xfs defaults 0 0
#/dev/mapper/centos-swap swap swap defaults 0 0
swapの行をコメントアウト。
(これ、kickstartインストールの時点で無効にできるのでは)
/usr/sbin/iptables-legacyがない。とりあえず飛ばそう…
firewalld
3/20追記: firewalldを有効にして以下の設定内容だとコンテナネットワークがうまく動いてない
使用ポートを通るようにする
master
$ sudo firewall-cmd --add-port=6443/tcp --permanent
$ sudo firewall-cmd --add-port=2379-2380/tcp --permanent
$ sudo firewall-cmd --add-port=10250/tcp --permanent
$ sudo firewall-cmd --add-port=10251/tcp --permanent
$ sudo firewall-cmd --add-port=10252/tcp --permanent
$ sudo firewall-cmd --reload
node
$ sudo firewall-cmd --add-port=10250/tcp --permanent
$ sudo firewall-cmd --add-port=30000-32767/tcp --permanent
$ sudo firewall-cmd --reload
共通: Flannelの場合は、8285/UDPと8472/UDPも追加
runtime
Docker CEを入れる
Get Docker Engine - Community for CentOS | Docker Documentation
$ sudo yum remove docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-engine
前提パッケージインストール
$ sudo yum install -y yum-utils device-mapper-persistent-data lvm2
リポジトリ追加
$ sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
インストール
$ sudo yum install docker-ce docker-ce-cli containerd.io
overlay2の設定追加
Use the OverlayFS storage driver | Docker Documentation
$ sudo mkdir -p /etc/docker
$ sudo vi /etc/docker/daemon.json
{
"storage-driver": "overlay2"
}
$ sudo systemctl start docker
$ sudo systemctl enable docker
[zaki@k8s-master01 ~]$ sudo docker info | grep -i storage
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Storage Driver: overlay2
なんか警告でとる。
kubeadmインストールのとこにこれを有効にする手順が書かれてるので、ここでやっておく
$ cat /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
$ sudo sysctl --system
警告も消えた。
kubeadm、kubelet、kubectlのインストール(master/worker共通)
kubeadm、kubelet、kubectlのインストール / kubeadmのインストール - Kubernetes
$ sudo sh -c "cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF"
石川さんにあやまる
手順がそうなってる…
OpenShiftだとenforce必須なんだけど。
$ sudo setenforce 0
$ sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
インストール
$ sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
enable
$ sudo systemctl enable --now kubelet
[zaki@k8s-master01 ~]$ systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: activating (auto-restart) (Result: exit-code) since 木 2020-03-19 17:11:13 JST; 1s ago
Docs: https://kubernetes.io/docs/
Process: 3161 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=255)
Main PID: 3161 (code=exited, status=255)
3月 19 17:11:13 k8s-master01.esxi.jp-z.jp systemd[1]: kubelet.service: main...
3月 19 17:11:13 k8s-master01.esxi.jp-z.jp systemd[1]: Unit kubelet.service ...
3月 19 17:11:13 k8s-master01.esxi.jp-z.jp systemd[1]: kubelet.service failed.
Hint: Some lines were ellipsized, use -l to show in full.
-- Unit kubelet.service has finished starting up.
--
-- The start-up result is done.
3月 19 17:11:03 k8s-master01.esxi.jp-z.jp kubelet[3152]: F0319 17:11:03.029999 3152 server.go:198] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml: no such file or directory
3月 19 17:11:03 k8s-master01.esxi.jp-z.jp systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
3月 19 17:11:03 k8s-master01.esxi.jp-z.jp systemd[1]: Unit kubelet.service entered failed state.
3月 19 17:11:03 k8s-master01.esxi.jp-z.jp systemd[1]: kubelet.service failed.
3月 19 17:11:13 k8s-master01.esxi.jp-z.jp systemd[1]: kubelet.service holdoff time over, scheduling restart.
3月 19 17:11:13 k8s-master01.esxi.jp-z.jp systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
なるほど、設定ファイルが無いと。
あ、でも、今の時点では起動できてなくてよいらしい。
masterノード構築
kubeadmを使用したシングルコントロールプレーンクラスターの作成 - Kubernetes
ノードOSで192.168.0.0/24を使ってるので、(手順にある192.168.0.0/16ではなく)172.30.0.0/16つかってkubeadm initしてみる。
$ sudo kubeadm init --pod-network-cidr=172.30.0.0/16
- 事前に
kubeadm config images pullやっとくといいっぽい
- アドレスは使用するCNIによって決まってるっぽい。
172.30.0.0/16なんて勝手にするのはイケてなさげ。やりなおそう
これは勝手に172.30.0.0.16を指定した例 (あとでやり直してる)
[zaki@k8s-master01 ~]$ sudo kubeadm init --pod-network-cidr=172.30.0.0/16
W0319 17:37:39.957971 13465 validation.go:28] Cannot validate kube-proxy config - no validator is available
W0319 17:37:39.958055 13465 validation.go:28] Cannot validate kubelet config - no validator is available
[init] Using Kubernetes version: v1.17.4
[preflight] Running pre-flight checks
[WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master01.esxi.jp-z.jp kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.121]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master01.esxi.jp-z.jp localhost] and IPs [192.168.0.121 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master01.esxi.jp-z.jp localhost] and IPs [192.168.0.121 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0319 17:38:16.215071 13465 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0319 17:38:16.215646 13465 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 33.502053 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.17" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master01.esxi.jp-z.jp as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master01.esxi.jp-z.jp as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: bx4ozo.aw1hsrh08bv101ed
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.0.121:6443 --token bx4ozo.aw1hsrh08bv101ed \
--discovery-token-ca-cert-hash sha256:bad635a8dfa329b6f9c308932891262b716f853dcae0aca5551933dfcb3c69bf
[zaki@k8s-master01 ~]$
お、うまくいった。
出力のkubeadm join ...の部分はノード追加時に使用するので記録しておくこと。
設定ファイル
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
[zaki@k8s-master01 ~]$ kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master01.esxi.jp-z.jp NotReady master 4m10s v1.17.4
この時点ではNotReady
[zaki@k8s-master01 ~]$ kubectl cluster-info
Kubernetes master is running at https://192.168.0.121:6443
KubeDNS is running at https://192.168.0.121:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.[zaki@k8s-master01 ~]$
アドレス再設定
前述の通り10.244.0.0/16でやりなおす。
やり直すまえにkubeadm resetで初期化する。
[zaki@k8s-master01 ~]$ sudo kubeadm reset
[reset] Reading configuration from the cluster...
[reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] Are you sure you want to proceed? [y/N]: y
[preflight] Running pre-flight checks
[reset] Removing info for node "k8s-master01.esxi.jp-z.jp" from the ConfigMap "kubeadm-config" in the "kube-system" Namespace
W0319 18:01:36.469586 20517 removeetcdmember.go:61] [reset] failed to remove etcd member: error syncing endpoints with etc: etcdclient: no available endpoints.Please manually remove this etcd member using etcdctl
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/etcd /var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni]
The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d
The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually by using the "iptables" command.
If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.
The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.
[zaki@k8s-master01 ~]$ rm -rf ~/.kube/
再kubeadm init
[zaki@k8s-master01 ~]$ sudo kubeadm init --pod-network-cidr=10.244.0.0/16
:
:
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.0.121:6443 --token gbe7un.jkmckrchkswwjoxm \
--discovery-token-ca-cert-hash sha256:580c73917da6033f5352c96b48ce681bd71057c90085f0b138a0e02a8963ef4f
設定ファイルのコピーと、kubeadm join ...の内容を確認。
[zaki@k8s-master01 ~]$ kubectl get pod -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system coredns-6955765f44-cxgvt 0/1 Pending 0 3m53s <none> <none> <none> <none>
kube-system coredns-6955765f44-zlvq8 0/1 Pending 0 3m53s <none> <none> <none> <none>
kube-system etcd-k8s-master01.esxi.jp-z.jp 1/1 Running 0 3m48s 192.168.0.121 k8s-master01.esxi.jp-z.jp <none> <none>
kube-system kube-apiserver-k8s-master01.esxi.jp-z.jp 1/1 Running 0 3m48s 192.168.0.121 k8s-master01.esxi.jp-z.jp <none> <none>
kube-system kube-controller-manager-k8s-master01.esxi.jp-z.jp 1/1 Running 0 3m48s 192.168.0.121 k8s-master01.esxi.jp-z.jp <none> <none>
kube-system kube-proxy-d99pk 1/1 Running 0 3m52s 192.168.0.121 k8s-master01.esxi.jp-z.jp <none> <none>
kube-system kube-scheduler-k8s-master01.esxi.jp-z.jp 1/1 Running 0 3m48s 192.168.0.121 k8s-master01.esxi.jp-z.jp <none> <none>
network設定がなくPendingになってる
podnetworkの設定
今回はFlanelを使ってみる。
Installing Addons - Kubernetes
$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml
[zaki@k8s-master01 ~]$ kubectl apply -f https://raw.githubusercontent.com/coreo
s/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.y
ml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds-amd64 created
daemonset.apps/kube-flannel-ds-arm64 created
daemonset.apps/kube-flannel-ds-arm created
daemonset.apps/kube-flannel-ds-ppc64le created
daemonset.apps/kube-flannel-ds-s390x created
[zaki@k8s-master01 ~]$ kubectl get pod -o wide -A
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system coredns-6955765f44-cxgvt 1/1 Running 0 5m 10.244.0.2 k8s-master01.esxi.jp-z.jp <none> <none>
kube-system coredns-6955765f44-zlvq8 1/1 Running 0 5m 10.244.0.3 k8s-master01.esxi.jp-z.jp <none> <none>
kube-system etcd-k8s-master01.esxi.jp-z.jp 1/1 Running 0 4m55s 192.168.0.121 k8s-master01.esxi.jp-z.jp <none> <none>
kube-system kube-apiserver-k8s-master01.esxi.jp-z.jp 1/1 Running 0 4m55s 192.168.0.121 k8s-master01.esxi.jp-z.jp <none> <none>
kube-system kube-controller-manager-k8s-master01.esxi.jp-z.jp 1/1 Running 0 4m55s 192.168.0.121 k8s-master01.esxi.jp-z.jp <none> <none>
kube-system kube-flannel-ds-amd64-jnc5m 1/1 Running 0 32s 192.168.0.121 k8s-master01.esxi.jp-z.jp <none> <none>
kube-system kube-proxy-d99pk 1/1 Running 0 4m59s 192.168.0.121 k8s-master01.esxi.jp-z.jp <none> <none>
kube-system kube-scheduler-k8s-master01.esxi.jp-z.jp 1/1 Running 0 4m55s 192.168.0.121 k8s-master01.esxi.jp-z.jp <none> <none>
うむ
[zaki@k8s-master01 ~]$ kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master01.esxi.jp-z.jp Ready master 14m v1.17.4
Readyになってた
コントロールプレーンノードの隔離
とりあえず飛ばす。
masterノードでアプリpodを動かすかどうかの設定。検証用なので動くようにしておいた方がリソース節約には良いと思うので、そのうち。
masterノードの追加
こちら。(たぶん初めからkubeadm initのオプションに、LBのアドレスを指定しておかなければならないっぽい)
zaki-hmkc.hatenablog.com
workerの構築
masterは動いたので、workerを追加する。
前述ミドルウェア設定と、kubeadm類のセットアップが完了したCentOS を用意。
master構築時に出力されたkubeadm join ...のコマンドをそのまま実行する。
$ sudo kubeadm join 192.168.0.121:6443 --token gbe7un.jkmckrchkswwjoxm \
--discovery-token-ca-cert-hash sha256:580c73917da6033f5352c96b48ce681bd71057c90085f0b138a0e02a8963ef4f
実行ログ
[zaki@k8s-node0101 ~]$ sudo kubeadm join 192.168.0.121:6443 --token gbe7un.jkmc
krchkswwjoxm \
> --discovery-token-ca-cert-hash sha256:580c73917da6033f5352c96b48ce681bd71
057c90085f0b138a0e02a8963ef4f
W0319 18:42:12.714978 10401 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Hostname]: hostname "k8s-node0101.esxi.jp-z.jp" could not be reached
[WARNING Hostname]: hostname "k8s-node0101.esxi.jp-z.jp": lookup k8s-node0101.esxi.jp-z.jp on 192.168.0.19:53: no such host
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.17" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
[zaki@k8s-node0101 ~]$
ほう
[zaki@k8s-master01 ~]$ kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master01.esxi.jp-z.jp Ready master 39m v1.17.4
k8s-node0101.esxi.jp-z.jp Ready <none> 46s v1.17.4
うごいたー
workerのROLESがnoneのままなのは正常？
コマンド/パラメタを失念した場合や期限が切れてる場合
zaki-hmkc.hatenablog.com
サンプルpodをデプロイ (疎通がうまくいってない)
[zaki@k8s-master01 ~]$ kubectl run web --image=httpd
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
deployment.apps/web created
[zaki@k8s-master01 ~]$ kubectl get pod
NAME READY STATUS RESTARTS AGE
web-7d6766b967-xf64h 1/1 Running 0 21s
うむ
[zaki@k8s-master01 ~]$ kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
web-7d6766b967-xf64h 1/1 Running 0 94s 10.244.1.2 k8s-node0101.esxi.jp-z.jp <none> <none>
[zaki@k8s-master01 ~]$ kubectl get svc,ep -l run=web
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/web ClusterIP 10.100.8.214 <none> 80/TCP 2m20s
NAME ENDPOINTS AGE
endpoints/web 10.244.1.2:80 2m20s
[zaki@k8s-master01 ~]$ curl 10.244.1.2:80
curl: (7) Failed connect to 10.244.1.2:80; 接続がタイムアウトしました
あれ？
podがデプロイされてるnode01からなら
[zaki@k8s-node0101 ~]$ curl 10.244.1.2:80
<html><body><h1>It works!</h1></body></html>
うごく。
クラスタのネットワークがちゃんと動いてないっぽい？(ノードを超えたpodにアクセスがうまくいってない)
NodePort設定でSerivceを作り直してみる
[zaki@k8s-master01 ~]$ kubectl expose deploy/web --port 80 --type NodePort
service/web exposed
[zaki@k8s-master01 ~]$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 111m
nginx ClusterIP 10.109.135.115 <none> 80/TCP 15m
web NodePort 10.96.171.215 <none> 80:30717/TCP 2s
[zaki@k8s-master01 ~]$ kubectl get svc -l run=web
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
web NodePort 10.96.171.215 <none> 80:30717/TCP 15s
[zaki@k8s-master01 ~]$ curl 192.168.0.125:30717
<html><body><h1>It works!</h1></body></html>
NodePortのServiceなら動くけど。。
要調査
(3/20追記) firewalldを無効にするとコンテナネットワークも動作
3/20追記
- まっさらな状態で(
kubeadm resetせずに)再構築: 変化なし
- Flanelのマニフェストをmasterバージョンに変更: 変化なし
- calicoに変更: ダメ
sudo kubeadm init --pod-network-cidr=192.168.0.0/16してkubectl apply -f https://docs.projectcalico.org/v3.8/manifests/calico.yaml: だめsudo kubeadm init --pod-network-cidr=10.244.0.0/16して、curlでローカルにマニフェストをDL,CALICO_IPV4POOL_CIDRのvalueを10.244.0.0/16に変更してapply: ダメ
Warning Unhealthy 48s kubelet, k8s-master01.esxi.jp-z.jp Readiness probe failed: calico/node is not ready: felix is not ready: readiness probe reporting 503
Warning Unhealthy 38s kubelet, k8s-master01.esxi.jp-z.jp Readiness probe failed: calico/node is not ready: BIRD is not ready: BGP not established with 192.168.0.1252020-03-19 21:34:21.405 [INFO][165] health.go 156: Number of node(s) with BGP peering established = 0
Warning Unhealthy 28s kubelet, k8s-master01.esxi.jp-z.jp Readiness probe failed: calico/node is not ready: BIRD is not ready: BGP not established with 192.168.0.1252020-03-19 21:34:31.402 [INFO][204] health.go 156: Number of node(s) with BGP peering established = 0
Warning Unhealthy 18s kubelet, k8s-master01.esxi.jp-z.jp Readiness probe failed: calico/node is not ready: BIRD is not ready: BGP not established with 192.168.0.1252020-03-19 21:34:41.399 [INFO][235] health.go 156: Number of node(s) with BGP peering established = 0
Warning Unhealthy 8s kubelet, k8s-master01.esxi.jp-z.jp Readiness probe failed: calico/node is not ready: BIRD is not ready: BGP not established with 192.168.0.1252020-03-19 21:34:51.400 [INFO][275] health.go 156: Number of node(s) with BGP peering established = 0
ん、、疎通できてない。。
firewalld周りかなぁ・・・ (でもss -anptしてもSYN_SENTとか出てこない)
必須ポートの確認
使用するPodネットワークプラグイン（以下を参照）のポートも開く必要があります。これは各Podネットワークプラグインによって異なるため、必要なポートについてはプラグインのドキュメントを参照してください。
あー
Flannelだと8285/udpと8472/udpが必要なので設定。
…したけど、timeoutしてたのがno route to hostになったけど、ノード跨ぎ・コンテナ間はまだ疎通できない。。(pingは行ける)
やっぱりどこかでフィルタリングかかってるっぽい。
切り分けのために、「最初から」firewalldをdisableにして再構築してみた。
[zaki@k8s-master01 ~]$ kubectl run apache --image=httpd
[zaki@k8s-master01 ~]$ kubectl expose deploy apache --port 80
[zaki@k8s-master01 ~]$ kubectl get pod -l run=apache -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
apache-7956695dc-bkb47 1/1 Running 0 7m49s 10.244.1.5 k8s-worker01.esxi.jp-z.jp <none> <none>
[zaki@k8s-master01 ~]$ curl http://10.244.1.5
<html><body><h1>It works!</h1></body></html>
うごいたわ。
[zaki@k8s-master01 ~]$ kubectl run centos --image=centos:7 -- tail -f /dev/null
[zaki@k8s-master01 ~]$ kubectl exec -it centos-6d6d6c7874-tmb67 bash
[root@centos-6d6d6c7874-tmb67 /]# curl apache
<html><body><h1>It works!</h1></body></html>
コンテナ間通信もいけた。
flannel.1 cni0インタフェースをtrustedゾーンに設定
Flannel入れた後に(入れる前でもOK)各ノードでこれ実行しておけば、「firewalldを有効にしたままコンテナ間通信」はひとまずできた。
$ sudo firewall-cmd --zone=trusted --change-interface=cni0
$ sudo firewall-cmd --reload
3/21追記: 最初flannel.1をtrusted設定にしてノード跨ぎでpodアクセス大丈夫だったけど、コンテナ間がダメだった。いろいろ試した感じだとcni0だと大丈夫だった
kubeadm init ...の実行前までを構築するPlaybook
github.com
