今回は(今回も)CentOS 7.7 1908 Minimalを素で入れて、公式ツールであるkubeadmを使ってKubernetesクラスタを作ってみた。
何だかんだで一度もやったことがなかった基礎…
追記2: masterを複数ノードにする場合は、kubeadm initに追加のオプションが必要っぽい
追記1: ちなみにクラスタはデプロイできたけど、firewalldが有効だとアプリケーションpodの通信がうまくいってないです。
- OSのインストール
- OS/ミドルウェア設定(master/worker共通)
- kubeadm、kubelet、kubectlのインストール(master/worker共通)
- masterノード構築
- workerの構築
- サンプルpodをデプロイ (疎通がうまくいってない)
- (3/20追記) firewalldを無効にするとコンテナネットワークも動作
- flannel.1 cni0インタフェースをtrustedゾーンに設定
OSのインストール
普通に。
ちなみにH/W要件は「2core」「RAM2GB(2GBの場合アプリの余裕無し)」とあるので、構成は以下の通り
| node | CPU | RAM |
|---|---|---|
| master | 4 | 8GB |
| worker | 4 | 8GB |
とりあえず1台ずつのみ。 また、ネットワーク内のDNS(dnsmasq使用)で、名前解決できるようになってる。
# k8s 192.168.0.121 k8s-master01.esxi.jp-z.jp 192.168.0.125 k8s-node01.esxi.jp-z.jp 192.168.0.121 k8s-master.esxi.jp-z.jp # LB 192.168.0.121 api.k8s.esxi.jp-z.jp # LB
マルチノードに備えて、LB用のFQDNも設定してる(今のところ出番なし)
OS/ミドルウェア設定(master/worker共通)
インストール後の状態
[zaki@k8s-master01 ~]$ cat /etc/redhat-release
CentOS Linux release 7.7.1908 (Core)
[zaki@k8s-master01 ~]$ df -h
ファイルシス サイズ 使用 残り 使用% マウント位置
devtmpfs 3.8G 0 3.8G 0% /dev
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 3.9G 12M 3.8G 1% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/mapper/centos-root 56G 1.2G 55G 3% /
/dev/sda1 1014M 150M 865M 15% /boot
tmpfs 781M 0 781M 0% /run/user/1000
[zaki@k8s-master01 ~]$ free -h
total used free shared buff/cache available
Mem: 7.6G 237M 7.2G 11M 203M 7.2G
Swap: 3.5G 0B 3.5G
swap off
[zaki@k8s-master01 ~]$ sudo swapoff -a
[zaki@k8s-master01 ~]$ free -h
total used free shared buff/cache available
Mem: 7.6G 235M 7.2G 11M 204M 7.2G
Swap: 0B 0B 0B
[zaki@k8s-master01 ~]$ cat /etc/fstab # # /etc/fstab # Created by anaconda on Thu Mar 19 06:59:37 2020 # # Accessible filesystems, by reference, are maintained under '/dev/disk' # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info # /dev/mapper/centos-root / xfs defaults 0 0 UUID=6a6cc094-d014-4870-b45a-96a3d2a858e4 /boot xfs defaults 0 0 #/dev/mapper/centos-swap swap swap defaults 0 0
swapの行をコメントアウト。
(これ、kickstartインストールの時点で無効にできるのでは)
iptablesのバックエンド
/usr/sbin/iptables-legacyがない。とりあえず飛ばそう…
firewalld
3/20追記: firewalldを有効にして以下の設定内容だとコンテナネットワークがうまく動いてない
使用ポートを通るようにする
master
$ sudo firewall-cmd --add-port=6443/tcp --permanent $ sudo firewall-cmd --add-port=2379-2380/tcp --permanent $ sudo firewall-cmd --add-port=10250/tcp --permanent $ sudo firewall-cmd --add-port=10251/tcp --permanent $ sudo firewall-cmd --add-port=10252/tcp --permanent $ sudo firewall-cmd --reload
node
$ sudo firewall-cmd --add-port=10250/tcp --permanent $ sudo firewall-cmd --add-port=30000-32767/tcp --permanent $ sudo firewall-cmd --reload
共通: Flannelの場合は、8285/UDPと8472/UDPも追加
runtime
Docker CEを入れる Get Docker Engine - Community for CentOS | Docker Documentation
$ sudo yum remove docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-engine
前提パッケージインストール
$ sudo yum install -y yum-utils device-mapper-persistent-data lvm2
リポジトリ追加
$ sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
インストール
$ sudo yum install docker-ce docker-ce-cli containerd.io
overlay2の設定追加 Use the OverlayFS storage driver | Docker Documentation
$ sudo mkdir -p /etc/docker $ sudo vi /etc/docker/daemon.json
{
"storage-driver": "overlay2"
}
$ sudo systemctl start docker $ sudo systemctl enable docker
[zaki@k8s-master01 ~]$ sudo docker info | grep -i storage WARNING: bridge-nf-call-iptables is disabled WARNING: bridge-nf-call-ip6tables is disabled Storage Driver: overlay2
なんか警告でとる。
kubeadmインストールのとこにこれを有効にする手順が書かれてるので、ここでやっておく
$ cat /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 $ sudo sysctl --system
警告も消えた。
kubeadm、kubelet、kubectlのインストール(master/worker共通)
kubeadm、kubelet、kubectlのインストール / kubeadmのインストール - Kubernetes
リポジトリ設定
$ sudo sh -c "cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg EOF"
石川さんにあやまる
手順がそうなってる…
OpenShiftだとenforce必須なんだけど。
$ sudo setenforce 0 $ sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
インストール
$ sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
enable
$ sudo systemctl enable --now kubelet
[zaki@k8s-master01 ~]$ systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: activating (auto-restart) (Result: exit-code) since 木 2020-03-19 17:11:13 JST; 1s ago
Docs: https://kubernetes.io/docs/
Process: 3161 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=255)
Main PID: 3161 (code=exited, status=255)
3月 19 17:11:13 k8s-master01.esxi.jp-z.jp systemd[1]: kubelet.service: main...
3月 19 17:11:13 k8s-master01.esxi.jp-z.jp systemd[1]: Unit kubelet.service ...
3月 19 17:11:13 k8s-master01.esxi.jp-z.jp systemd[1]: kubelet.service failed.
Hint: Some lines were ellipsized, use -l to show in full.
-- Unit kubelet.service has finished starting up. -- -- The start-up result is done. 3月 19 17:11:03 k8s-master01.esxi.jp-z.jp kubelet[3152]: F0319 17:11:03.029999 3152 server.go:198] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml: no such file or directory 3月 19 17:11:03 k8s-master01.esxi.jp-z.jp systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a 3月 19 17:11:03 k8s-master01.esxi.jp-z.jp systemd[1]: Unit kubelet.service entered failed state. 3月 19 17:11:03 k8s-master01.esxi.jp-z.jp systemd[1]: kubelet.service failed. 3月 19 17:11:13 k8s-master01.esxi.jp-z.jp systemd[1]: kubelet.service holdoff time over, scheduling restart. 3月 19 17:11:13 k8s-master01.esxi.jp-z.jp systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
なるほど、設定ファイルが無いと。
あ、でも、今の時点では起動できてなくてよいらしい。
masterノード構築
kubeadmを使用したシングルコントロールプレーンクラスターの作成 - Kubernetes
ノードOSで192.168.0.0/24を使ってるので、(手順にある192.168.0.0/16ではなく)172.30.0.0/16つかってkubeadm initしてみる。
$ sudo kubeadm init --pod-network-cidr=172.30.0.0/16
- 事前に
kubeadm config images pullやっとくといいっぽい - アドレスは使用するCNIによって決まってるっぽい。
172.30.0.0/16なんて勝手にするのはイケてなさげ。やりなおそう- Calicoだと、マニフェストに192.168.0.0/16と記述がある
- まぁ書き換えればいいんだろうけど影響範囲がわからん
- Flannelだと10.244.0.0/16を指定するようなので、これを使ってみる
- Calicoだと、マニフェストに192.168.0.0/16と記述がある
これは勝手に172.30.0.0.16を指定した例 (あとでやり直してる)
[zaki@k8s-master01 ~]$ sudo kubeadm init --pod-network-cidr=172.30.0.0/16
W0319 17:37:39.957971 13465 validation.go:28] Cannot validate kube-proxy config - no validator is available
W0319 17:37:39.958055 13465 validation.go:28] Cannot validate kubelet config - no validator is available
[init] Using Kubernetes version: v1.17.4
[preflight] Running pre-flight checks
[WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master01.esxi.jp-z.jp kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.121]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master01.esxi.jp-z.jp localhost] and IPs [192.168.0.121 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master01.esxi.jp-z.jp localhost] and IPs [192.168.0.121 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0319 17:38:16.215071 13465 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0319 17:38:16.215646 13465 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 33.502053 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.17" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master01.esxi.jp-z.jp as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master01.esxi.jp-z.jp as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: bx4ozo.aw1hsrh08bv101ed
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.0.121:6443 --token bx4ozo.aw1hsrh08bv101ed \
--discovery-token-ca-cert-hash sha256:bad635a8dfa329b6f9c308932891262b716f853dcae0aca5551933dfcb3c69bf
[zaki@k8s-master01 ~]$
お、うまくいった。
出力のkubeadm join ...の部分はノード追加時に使用するので記録しておくこと。
設定ファイル
$ mkdir -p $HOME/.kube $ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config $ sudo chown $(id -u):$(id -g) $HOME/.kube/config
[zaki@k8s-master01 ~]$ kubectl get node NAME STATUS ROLES AGE VERSION k8s-master01.esxi.jp-z.jp NotReady master 4m10s v1.17.4
この時点ではNotReady
[zaki@k8s-master01 ~]$ kubectl cluster-info Kubernetes master is running at https://192.168.0.121:6443 KubeDNS is running at https://192.168.0.121:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.[zaki@k8s-master01 ~]$
アドレス再設定
前述の通り10.244.0.0/16でやりなおす。
やり直すまえにkubeadm resetで初期化する。
[zaki@k8s-master01 ~]$ sudo kubeadm reset [reset] Reading configuration from the cluster... [reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted. [reset] Are you sure you want to proceed? [y/N]: y [preflight] Running pre-flight checks [reset] Removing info for node "k8s-master01.esxi.jp-z.jp" from the ConfigMap "kubeadm-config" in the "kube-system" Namespace W0319 18:01:36.469586 20517 removeetcdmember.go:61] [reset] failed to remove etcd member: error syncing endpoints with etc: etcdclient: no available endpoints.Please manually remove this etcd member using etcdctl [reset] Stopping the kubelet service [reset] Unmounting mounted directories in "/var/lib/kubelet" [reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki] [reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf] [reset] Deleting contents of stateful directories: [/var/lib/etcd /var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni] The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d The reset process does not reset or clean up iptables rules or IPVS tables. If you wish to reset iptables, you must do so manually by using the "iptables" command. If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar) to reset your system's IPVS tables. The reset process does not clean your kubeconfig files and you must remove them manually. Please, check the contents of the $HOME/.kube/config file. [zaki@k8s-master01 ~]$ rm -rf ~/.kube/
再kubeadm init
[zaki@k8s-master01 ~]$ sudo kubeadm init --pod-network-cidr=10.244.0.0/16
:
:
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.0.121:6443 --token gbe7un.jkmckrchkswwjoxm \
--discovery-token-ca-cert-hash sha256:580c73917da6033f5352c96b48ce681bd71057c90085f0b138a0e02a8963ef4f
設定ファイルのコピーと、kubeadm join ...の内容を確認。
[zaki@k8s-master01 ~]$ kubectl get pod -A -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system coredns-6955765f44-cxgvt 0/1 Pending 0 3m53s <none> <none> <none> <none> kube-system coredns-6955765f44-zlvq8 0/1 Pending 0 3m53s <none> <none> <none> <none> kube-system etcd-k8s-master01.esxi.jp-z.jp 1/1 Running 0 3m48s 192.168.0.121 k8s-master01.esxi.jp-z.jp <none> <none> kube-system kube-apiserver-k8s-master01.esxi.jp-z.jp 1/1 Running 0 3m48s 192.168.0.121 k8s-master01.esxi.jp-z.jp <none> <none> kube-system kube-controller-manager-k8s-master01.esxi.jp-z.jp 1/1 Running 0 3m48s 192.168.0.121 k8s-master01.esxi.jp-z.jp <none> <none> kube-system kube-proxy-d99pk 1/1 Running 0 3m52s 192.168.0.121 k8s-master01.esxi.jp-z.jp <none> <none> kube-system kube-scheduler-k8s-master01.esxi.jp-z.jp 1/1 Running 0 3m48s 192.168.0.121 k8s-master01.esxi.jp-z.jp <none> <none>
network設定がなくPendingになってる
podnetworkの設定
今回はFlanelを使ってみる。
Installing Addons - Kubernetes
$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml
[zaki@k8s-master01 ~]$ kubectl apply -f https://raw.githubusercontent.com/coreo s/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.y ml podsecuritypolicy.policy/psp.flannel.unprivileged created clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel created serviceaccount/flannel created configmap/kube-flannel-cfg created daemonset.apps/kube-flannel-ds-amd64 created daemonset.apps/kube-flannel-ds-arm64 created daemonset.apps/kube-flannel-ds-arm created daemonset.apps/kube-flannel-ds-ppc64le created daemonset.apps/kube-flannel-ds-s390x created
[zaki@k8s-master01 ~]$ kubectl get pod -o wide -A NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system coredns-6955765f44-cxgvt 1/1 Running 0 5m 10.244.0.2 k8s-master01.esxi.jp-z.jp <none> <none> kube-system coredns-6955765f44-zlvq8 1/1 Running 0 5m 10.244.0.3 k8s-master01.esxi.jp-z.jp <none> <none> kube-system etcd-k8s-master01.esxi.jp-z.jp 1/1 Running 0 4m55s 192.168.0.121 k8s-master01.esxi.jp-z.jp <none> <none> kube-system kube-apiserver-k8s-master01.esxi.jp-z.jp 1/1 Running 0 4m55s 192.168.0.121 k8s-master01.esxi.jp-z.jp <none> <none> kube-system kube-controller-manager-k8s-master01.esxi.jp-z.jp 1/1 Running 0 4m55s 192.168.0.121 k8s-master01.esxi.jp-z.jp <none> <none> kube-system kube-flannel-ds-amd64-jnc5m 1/1 Running 0 32s 192.168.0.121 k8s-master01.esxi.jp-z.jp <none> <none> kube-system kube-proxy-d99pk 1/1 Running 0 4m59s 192.168.0.121 k8s-master01.esxi.jp-z.jp <none> <none> kube-system kube-scheduler-k8s-master01.esxi.jp-z.jp 1/1 Running 0 4m55s 192.168.0.121 k8s-master01.esxi.jp-z.jp <none> <none>
うむ
[zaki@k8s-master01 ~]$ kubectl get node NAME STATUS ROLES AGE VERSION k8s-master01.esxi.jp-z.jp Ready master 14m v1.17.4
Readyになってた
コントロールプレーンノードの隔離
とりあえず飛ばす。
masterノードでアプリpodを動かすかどうかの設定。検証用なので動くようにしておいた方がリソース節約には良いと思うので、そのうち。
masterノードの追加
こちら。(たぶん初めからkubeadm initのオプションに、LBのアドレスを指定しておかなければならないっぽい)
workerの構築
masterは動いたので、workerを追加する。
前述ミドルウェア設定と、kubeadm類のセットアップが完了したCentOS を用意。
master構築時に出力されたkubeadm join ...のコマンドをそのまま実行する。
$ sudo kubeadm join 192.168.0.121:6443 --token gbe7un.jkmckrchkswwjoxm \
--discovery-token-ca-cert-hash sha256:580c73917da6033f5352c96b48ce681bd71057c90085f0b138a0e02a8963ef4f
実行ログ
[zaki@k8s-node0101 ~]$ sudo kubeadm join 192.168.0.121:6443 --token gbe7un.jkmc
krchkswwjoxm \
> --discovery-token-ca-cert-hash sha256:580c73917da6033f5352c96b48ce681bd71
057c90085f0b138a0e02a8963ef4f
W0319 18:42:12.714978 10401 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Hostname]: hostname "k8s-node0101.esxi.jp-z.jp" could not be reached
[WARNING Hostname]: hostname "k8s-node0101.esxi.jp-z.jp": lookup k8s-node0101.esxi.jp-z.jp on 192.168.0.19:53: no such host
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.17" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
[zaki@k8s-node0101 ~]$
ほう
[zaki@k8s-master01 ~]$ kubectl get node NAME STATUS ROLES AGE VERSION k8s-master01.esxi.jp-z.jp Ready master 39m v1.17.4 k8s-node0101.esxi.jp-z.jp Ready <none> 46s v1.17.4
うごいたー
workerのROLESがnoneのままなのは正常？
コマンド/パラメタを失念した場合や期限が切れてる場合
サンプルpodをデプロイ (疎通がうまくいってない)
[zaki@k8s-master01 ~]$ kubectl run web --image=httpd kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead. deployment.apps/web created [zaki@k8s-master01 ~]$ kubectl get pod NAME READY STATUS RESTARTS AGE web-7d6766b967-xf64h 1/1 Running 0 21s
うむ
[zaki@k8s-master01 ~]$ kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES web-7d6766b967-xf64h 1/1 Running 0 94s 10.244.1.2 k8s-node0101.esxi.jp-z.jp <none> <none>
[zaki@k8s-master01 ~]$ kubectl get svc,ep -l run=web NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/web ClusterIP 10.100.8.214 <none> 80/TCP 2m20s NAME ENDPOINTS AGE endpoints/web 10.244.1.2:80 2m20s
[zaki@k8s-master01 ~]$ curl 10.244.1.2:80 curl: (7) Failed connect to 10.244.1.2:80; 接続がタイムアウトしました
あれ？
podがデプロイされてるnode01からなら
[zaki@k8s-node0101 ~]$ curl 10.244.1.2:80 <html><body><h1>It works!</h1></body></html>
うごく。
クラスタのネットワークがちゃんと動いてないっぽい？(ノードを超えたpodにアクセスがうまくいってない)
NodePort設定でSerivceを作り直してみる
[zaki@k8s-master01 ~]$ kubectl expose deploy/web --port 80 --type NodePort service/web exposed [zaki@k8s-master01 ~]$ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 111m nginx ClusterIP 10.109.135.115 <none> 80/TCP 15m web NodePort 10.96.171.215 <none> 80:30717/TCP 2s [zaki@k8s-master01 ~]$ kubectl get svc -l run=web NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE web NodePort 10.96.171.215 <none> 80:30717/TCP 15s
[zaki@k8s-master01 ~]$ curl 192.168.0.125:30717 <html><body><h1>It works!</h1></body></html>
NodePortのServiceなら動くけど。。
要調査
(3/20追記) firewalldを無効にするとコンテナネットワークも動作
3/20追記
- まっさらな状態で(
kubeadm resetせずに)再構築: 変化なし - Flanelのマニフェストをmasterバージョンに変更: 変化なし
- calicoに変更: ダメ
sudo kubeadm init --pod-network-cidr=192.168.0.0/16してkubectl apply -f https://docs.projectcalico.org/v3.8/manifests/calico.yaml: だめsudo kubeadm init --pod-network-cidr=10.244.0.0/16して、curlでローカルにマニフェストをDL,CALICO_IPV4POOL_CIDRのvalueを10.244.0.0/16に変更してapply: ダメ
Warning Unhealthy 48s kubelet, k8s-master01.esxi.jp-z.jp Readiness probe failed: calico/node is not ready: felix is not ready: readiness probe reporting 503 Warning Unhealthy 38s kubelet, k8s-master01.esxi.jp-z.jp Readiness probe failed: calico/node is not ready: BIRD is not ready: BGP not established with 192.168.0.1252020-03-19 21:34:21.405 [INFO][165] health.go 156: Number of node(s) with BGP peering established = 0 Warning Unhealthy 28s kubelet, k8s-master01.esxi.jp-z.jp Readiness probe failed: calico/node is not ready: BIRD is not ready: BGP not established with 192.168.0.1252020-03-19 21:34:31.402 [INFO][204] health.go 156: Number of node(s) with BGP peering established = 0 Warning Unhealthy 18s kubelet, k8s-master01.esxi.jp-z.jp Readiness probe failed: calico/node is not ready: BIRD is not ready: BGP not established with 192.168.0.1252020-03-19 21:34:41.399 [INFO][235] health.go 156: Number of node(s) with BGP peering established = 0 Warning Unhealthy 8s kubelet, k8s-master01.esxi.jp-z.jp Readiness probe failed: calico/node is not ready: BIRD is not ready: BGP not established with 192.168.0.1252020-03-19 21:34:51.400 [INFO][275] health.go 156: Number of node(s) with BGP peering established = 0
ん、、疎通できてない。。
firewalld周りかなぁ・・・ (でもss -anptしてもSYN_SENTとか出てこない)
使用するPodネットワークプラグイン（以下を参照）のポートも開く必要があります。これは各Podネットワークプラグインによって異なるため、必要なポートについてはプラグインのドキュメントを参照してください。
あー
Flannelだと8285/udpと8472/udpが必要なので設定。
…したけど、timeoutしてたのがno route to hostになったけど、ノード跨ぎ・コンテナ間はまだ疎通できない。。(pingは行ける)
やっぱりどこかでフィルタリングかかってるっぽい。
切り分けのために、「最初から」firewalldをdisableにして再構築してみた。
[zaki@k8s-master01 ~]$ kubectl run apache --image=httpd [zaki@k8s-master01 ~]$ kubectl expose deploy apache --port 80 [zaki@k8s-master01 ~]$ kubectl get pod -l run=apache -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES apache-7956695dc-bkb47 1/1 Running 0 7m49s 10.244.1.5 k8s-worker01.esxi.jp-z.jp <none> <none> [zaki@k8s-master01 ~]$ curl http://10.244.1.5 <html><body><h1>It works!</h1></body></html>
うごいたわ。
[zaki@k8s-master01 ~]$ kubectl run centos --image=centos:7 -- tail -f /dev/null [zaki@k8s-master01 ~]$ kubectl exec -it centos-6d6d6c7874-tmb67 bash [root@centos-6d6d6c7874-tmb67 /]# curl apache <html><body><h1>It works!</h1></body></html>
コンテナ間通信もいけた。
flannel.1 cni0インタフェースをtrustedゾーンに設定
Flannel入れた後に(入れる前でもOK)各ノードでこれ実行しておけば、「firewalldを有効にしたままコンテナ間通信」はひとまずできた。
$ sudo firewall-cmd --zone=trusted --change-interface=cni0 $ sudo firewall-cmd --reload
3/21追記: 最初flannel.1をtrusted設定にしてノード跨ぎでpodアクセス大丈夫だったけど、コンテナ間がダメだった。いろいろ試した感じだとcni0だと大丈夫だった
kubeadm init ...の実行前までを構築するPlaybook
