zaki work log

作業ログやら生活ログやらなんやら

[Kubernetes / Istio] Sock Shopの現k8sバージョン対応とIstio対応デプロイお試し

microservices-demo.github.io

環境

[zaki@cloud-dev ~]$ kc version --short
Client Version: v1.18.8
Server Version: v1.18.5

みんな大好きkubeadmで作ったオンプレK8s(MetalLBによるLoadbalancer Service対応)です。

[zaki@cloud-dev ~]$ istioctl version
client version: 1.6.5
control plane version: 1.6.5
data plane version: 1.6.5 (22 proxies)

Istioバージョンは最新追えていなくて1.6.5。

quickstart

とりあえずデプロイしてみるぞ!
https://microservices-demo.github.io/docs/quickstart.html

と言っても、まずGitのソースをcloneします。

$ git clone https://github.com/microservices-demo/microservices-demo
cd microservices-demo

ただし、quickstartの記載はdocker-composeの場合なので、Any Kubernetes Clusterを参照。

まずnamespaceを作成

$ kubectl create namespace sock-shop

そして、デプロイ用マニフェストapplyする。
マニフェスト見る限り、namespaceのデプロイ先はsock-shop固定になっている。

$ kubectl apply -f deploy/kubernetes/complete-demo.yaml
[zaki@cloud-dev microservices-demo]$ kubectl apply -f deploy/kubernetes/complete-demo.yaml
service/carts-db created
service/carts created
service/catalogue-db created
service/catalogue created
service/front-end created
service/orders-db created
service/orders created
service/payment created
service/queue-master created
service/rabbitmq created
service/shipping created
service/user-db created
service/user created
unable to recognize "deploy/kubernetes/complete-demo.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "deploy/kubernetes/complete-demo.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "deploy/kubernetes/complete-demo.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "deploy/kubernetes/complete-demo.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "deploy/kubernetes/complete-demo.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "deploy/kubernetes/complete-demo.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "deploy/kubernetes/complete-demo.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "deploy/kubernetes/complete-demo.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "deploy/kubernetes/complete-demo.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "deploy/kubernetes/complete-demo.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "deploy/kubernetes/complete-demo.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "deploy/kubernetes/complete-demo.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "deploy/kubernetes/complete-demo.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"

あらら、DeploymentリソースのapiVersionextensions/v1beta1になっている。
これ、Kubernetes 1.16からはapps/v1にする必要があるので要修正。(後述)

diff --git a/deploy/kubernetes/complete-demo.yaml b/deploy/kubernetes/complete-demo.yaml
index a11f99b..64ac9c0 100644
--- a/deploy/kubernetes/complete-demo.yaml
+++ b/deploy/kubernetes/complete-demo.yaml
@@ -1 +1 @@
-apiVersion: extensions/v1beta1
+apiVersion: apps/v1
@@ -55 +55 @@ spec:
-apiVersion: extensions/v1beta1
+apiVersion: apps/v1

[...]

※ 同じ修正の繰り返しなので省略

これで再デプロイ

…をすると、今度はselectorのエラー

$ kubectl apply -f deploy/kubernetes/complete-demo.yaml
error: error validating "deploy/kubernetes/complete-demo.yaml": error validating data: ValidationError(Deployment.spec): missing required field "selector" in io.k8s.api.apps.v1.DeploymentSpec; if you choose to ignore these errors, turn validation off with --validate=false

specにはselectorが必要ということ。
前述のextensions/v1beta1の変更と合わせて、修正内容は以下の通り。

github.com

これでデプロイ

[zaki@cloud-dev microservices-demo]$ kubectl apply -f deploy/kubernetes/complete-demo.yaml
deployment.apps/carts-db created
service/carts-db unchanged
deployment.apps/carts created
service/carts unchanged
deployment.apps/catalogue-db created
service/catalogue-db unchanged
deployment.apps/catalogue created
service/catalogue unchanged
deployment.apps/front-end created
service/front-end unchanged
deployment.apps/orders-db created
service/orders-db unchanged
deployment.apps/orders created
service/orders unchanged
deployment.apps/payment created
service/payment unchanged
deployment.apps/queue-master created
service/queue-master unchanged
deployment.apps/rabbitmq created
service/rabbitmq unchanged
deployment.apps/shipping created
service/shipping unchanged
deployment.apps/user-db created
service/user-db unchanged
deployment.apps/user created
service/user unchanged

とりあえずデプロイは成功。
しばらく待てば、各podともにRunningになるはず。

[zaki@cloud-dev microservices-demo]$ kc get pod,svc -n sock-shop
NAME                                READY   STATUS    RESTARTS   AGE
pod/carts-7bbbd7779d-bwwm2          1/1     Running   0          8m9s
pod/carts-db-84b777d9c-nzt8m        1/1     Running   0          8m9s
pod/catalogue-8684f655d9-m9dxk      1/1     Running   0          8m9s
pod/catalogue-db-5579f7f4cb-55gtk   1/1     Running   0          8m9s
pod/front-end-6f5fc69d6-brlb8       1/1     Running   0          8m9s
pod/orders-7ccf68495-p599t          1/1     Running   0          8m9s
pod/orders-db-77c46b9c85-m5q55      1/1     Running   0          8m9s
pod/payment-74976d749f-ktmp7        1/1     Running   0          8m9s
pod/queue-master-799b6b57d5-bc9pt   1/1     Running   0          8m9s
pod/rabbitmq-8458d7d4c-2zwlf        1/1     Running   0          8m9s
pod/shipping-5c89f4886c-8pppb       1/1     Running   0          8m9s
pod/user-d6c48f668-wqs2d            1/1     Running   0          8m8s
pod/user-db-5cdfd6d44b-26w5j        1/1     Running   0          8m8s

NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
service/carts          ClusterIP   10.96.85.13      <none>        80/TCP         25m
service/carts-db       ClusterIP   10.108.236.130   <none>        27017/TCP      25m
service/catalogue      ClusterIP   10.110.18.120    <none>        80/TCP         25m
service/catalogue-db   ClusterIP   10.97.198.131    <none>        3306/TCP       25m
service/front-end      NodePort    10.101.204.208   <none>        80:30001/TCP   25m
service/orders         ClusterIP   10.103.82.66     <none>        80/TCP         25m
service/orders-db      ClusterIP   10.104.77.171    <none>        27017/TCP      25m
service/payment        ClusterIP   10.107.33.54     <none>        80/TCP         25m
service/queue-master   ClusterIP   10.100.115.94    <none>        80/TCP         25m
service/rabbitmq       ClusterIP   10.110.36.96     <none>        5672/TCP       25m
service/shipping       ClusterIP   10.107.24.182    <none>        80/TCP         25m
service/user           ClusterIP   10.101.65.142    <none>        80/TCP         25m
service/user-db        ClusterIP   10.101.67.24     <none>        27017/TCP      25m

webアクセス

NodePortのServiceもデプロイされているので、ノードにアクセスする。

[zaki@cloud-dev microservices-demo]$ kc get svc -n sock-shop  front-end
NAME        TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
front-end   NodePort   10.101.204.208   <none>        80:30001/TCP   36m
[zaki@cloud-dev microservices-demo]$ kc get node -o wide
NAME                        STATUS   ROLES    AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION                CONTAINER-RUNTIME
k8s-master01.esxi.jp-z.jp   Ready    master   53d   v1.18.5   192.168.0.121   <none>        CentOS Linux 7 (Core)   3.10.0-1127.13.1.el7.x86_64   docker://19.3.12
k8s-worker01.esxi.jp-z.jp   Ready    <none>   53d   v1.18.5   192.168.0.125   <none>        CentOS Linux 7 (Core)   3.10.0-1127.13.1.el7.x86_64   docker://19.3.12
k8s-worker02.esxi.jp-z.jp   Ready    <none>   53d   v1.18.5   192.168.0.126   <none>        CentOS Linux 7 (Core)   3.10.0-1127.13.1.el7.x86_64   docker://19.3.12

この場合であれば3台のノードのうちどれでも任意のノードの30001/TCPにアクセスすればOK

f:id:zaki-hmkc:20200826214127p:plain

※ NodePort Serviceを使っているので、URLがノードのアドレス:NodePortのポートであるhttp://192.168.0.125:30001/になっている。


apiVersionextensions/v1beta1については、こちらのKubernetes 1.15 -> 1.16の変更点を参照。

qiita.com

Istio対応

サイドカーインジェクション用namespace

Istio上で動かすにはistio-injection=enabledというラベルが設定されたnamespaceにデプロイする。
これだけ!

[zaki@cloud-dev microservices-demo]$ kc create namespace sock-shop-mesh
namespace/sock-shop-mesh created
[zaki@cloud-dev microservices-demo]$ kc label namespace sock-shop-mesh istio-injection=enabled
namespace/sock-shop-mesh labeled

で、Sock Shopデプロイ用マニフェストから(任意のネームスペースにデプロイできるように)全てのnamespace: sock-shopを削除して、sock-shop-meshネームスペースを指定してデプロイ

github.com

↓はgrep -vで雑に除外してるけど、↑に修正版マニフェスト置いています。

[zaki@cloud-dev microservices-demo]$ grep -v namespace deploy/kubernetes/complete-demo.yaml | kubectl apply -f - -n
sock-shop-mesh
deployment.apps/carts-db created
service/carts-db created
deployment.apps/carts created
service/carts created
deployment.apps/catalogue-db created
service/catalogue-db created
deployment.apps/catalogue created
service/catalogue created
deployment.apps/front-end created
deployment.apps/orders-db created
service/orders-db created
deployment.apps/orders created
service/orders created
deployment.apps/payment created
service/payment created
deployment.apps/queue-master created
service/queue-master created
deployment.apps/rabbitmq created
service/rabbitmq created
deployment.apps/shipping created
service/shipping created
deployment.apps/user-db created
service/user-db created
deployment.apps/user created
service/user created
The Service "front-end" is invalid: spec.ports[0].nodePort: Invalid value: 30001: provided port is already allocated

おっと、NodePortのポート番号指定してたのか。とりあえず無視…はNodePortとしては不要だけど、Service自体は(外部からのアクセス時のルーティング先として)必要なので定義を変更してClusterIPで再デプロイ。

github.com

[zaki@cloud-dev microservices-demo]$ grep -v namespace deploy/kubernetes/complete-demo.yaml | kubectl apply -f - -n
sock-shop-mesh
deployment.apps/carts-db unchanged
service/carts-db unchanged
deployment.apps/carts unchanged
service/carts unchanged
deployment.apps/catalogue-db unchanged
service/catalogue-db unchanged
deployment.apps/catalogue unchanged
service/catalogue unchanged
deployment.apps/front-end unchanged
service/front-end created                ### <- これ
deployment.apps/orders-db unchanged
service/orders-db unchanged
deployment.apps/orders unchanged
service/orders unchanged
deployment.apps/payment unchanged
service/payment unchanged
deployment.apps/queue-master unchanged
service/queue-master unchanged
deployment.apps/rabbitmq unchanged
service/rabbitmq unchanged
deployment.apps/shipping unchanged
service/shipping unchanged
deployment.apps/user-db unchanged
service/user-db unchanged
deployment.apps/user unchanged
service/user unchanged
[zaki@cloud-dev microservices-demo]$ kc get pod -n sock-shop-mesh
NAME                            READY   STATUS            RESTARTS   AGE
carts-7bbbd7779d-hvvzf          2/2     Running           0          70s
carts-db-84b777d9c-mptnz        2/2     Running           0          70s
catalogue-8684f655d9-gf6xn      0/2     PodInitializing   0          69s
catalogue-db-5579f7f4cb-xqf48   2/2     Running           0          70s
front-end-6f5fc69d6-dbqxn       0/2     PodInitializing   0          69s
orders-7ccf68495-tgwqv          0/2     PodInitializing   0          69s
orders-db-77c46b9c85-5k4zg      2/2     Running           0          69s
payment-74976d749f-wfwjn        0/2     PodInitializing   0          68s
queue-master-799b6b57d5-l4gq8   2/2     Running           0          68s
rabbitmq-8458d7d4c-cbhcx        2/2     Running           0          68s
shipping-5c89f4886c-m4s4n       0/2     PodInitializing   0          68s
user-d6c48f668-krwgr            0/2     PodInitializing   0          68s
user-db-5cdfd6d44b-gjdg4        0/2     PodInitializing   0          68s

この通り、コンテナ数が2/2になっており、サイドカーのistio-proxyのインジェクションが行われている。

[zaki@cloud-dev microservices-demo]$ kc get pod -n sock-shop-mesh
NAME                            READY   STATUS    RESTARTS   AGE
carts-7bbbd7779d-hvvzf          2/2     Running   0          2m15s
carts-db-84b777d9c-mptnz        2/2     Running   0          2m15s
catalogue-8684f655d9-gf6xn      2/2     Running   0          2m14s
catalogue-db-5579f7f4cb-xqf48   2/2     Running   0          2m15s
front-end-6f5fc69d6-dbqxn       2/2     Running   0          2m14s
orders-7ccf68495-tgwqv          2/2     Running   0          2m14s
orders-db-77c46b9c85-5k4zg      2/2     Running   0          2m14s
payment-74976d749f-wfwjn        2/2     Running   0          2m13s
queue-master-799b6b57d5-l4gq8   2/2     Running   0          2m13s
rabbitmq-8458d7d4c-cbhcx        2/2     Running   0          2m13s
shipping-5c89f4886c-m4s4n       2/2     Running   0          2m13s
user-d6c48f668-krwgr            2/2     Running   0          2m13s
user-db-5cdfd6d44b-gjdg4        2/2     Running   0          2m13s

デプロイ完了。

Gateway/VirtualService

Istio用のサイドカー込みのアプリケーションpodはデプロイできたので、次は外部アクセスに必要なリソースを作成。

最低限必要なのは、GatewayリソースとVirtualServiceリソース、それと、Istio IngressGateway経由でアクセスする際に使用するドメイン。(今回はIngress的にほかのアプリケーション用の定義との重複条件が無いので無くても良い:hosts: "*"としている)

GatewayとVirtualServiceはBookInfoの設定を参考。

Gateway

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: sock-shop
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:  # 外部からのIstio IngressGatewayへのアクセスについての定義を記述
      name: http
      number: 80
      protocol: HTTP

VirtualService

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: sock-shop
spec:
  gateways:
  - sock-shop    # 上記の対象Gatewayリソース名
  hosts:
  - '*'
  http:
  - route:
    - destination:      # このリソースが受けたトラフィックをどのServiceへ転送するかを指定(これがNodePortは不要だけどServiceが必要と書いた部分)
        host: front-end
        port:
          number: 80

ソースはこちら

github.com

デプロイして実行

[zaki@cloud-dev sock-shop]$ kc apply -f sockshop-gateway.yaml -n sock-shop-mesh
gateway.networking.istio.io/sock-shop created
virtualservice.networking.istio.io/bsock-shop created
[zaki@cloud-dev sock-shop]$ kc get svc -n istio-system istio-ingressgateway
NAME                   TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)
                             AGE
istio-ingressgateway   LoadBalancer   10.104.82.61   192.168.0.182   15020:31343/TCP,80:30446/TCP,443:31013/TCP,31400:32549/TCP,15443:31115/TCP   6d1h

オリジナルのpodはNodePortを使ったけれど、Istio上のpodへのアクセスはIstio IngressGatewayを使用する。

f:id:zaki-hmkc:20200826222432p:plain

※ NodePort Serviceでなく、Istio IngressGatewayを使っているので、Istio未使用版のURLと異なっている。

Kialiでトラフィックを確認

Istioで動いてるので、Kiali使って状態を確認もできる。

f:id:zaki-hmkc:20200826222818p:plain

Kialiはこちらの記事もどぞ。

zaki-hmkc.hatenablog.com


春から初夏まで仕事でIstioを ちょっとだけ 触ってて(ほぼトラブルシューティング)、もっと基礎力を上げたくて何か良い題材がないかツイートしたところ、OpenShift環境でSock Shopをアレンジしている方からリプ頂いていたのに、なかなか時間取れずに間が空いてしまいましたが、2020年8月のKubernetes Meetup Tokyo #33でもSock ShopとKialiの話が出てきて、今こそやらねば!と、ひとまずコミュニティ版IstioでSock Shop対応してみました。


Sock Shopを使ったデプロイ例は、Kubernetes実践ガイドでも扱っているので興味ある人は是非~ (ステマ)