[08] 為什麼 EKS 可以整合 IAM roles for service accounts（IRSA）（一）

根據 EKS IAM roles for service accounts¹，我們可以透過於在 Kubernetes 上的 Service Account 及 Pod annotation 設置 IAM role ARN 後，則 Pod 就可以取得此 IAM role 對應的權限。

因此本文希望探討了解「為什麼 EKS 是如何根據原生 Kubernetes 功能，整合 IAM 權限最終使得 Pod 內環境可以獲取此 IAM role」。

建置環境

建立以下 IAM policy 並儲存為 iam-policy.json JSON。

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject",
                "s3:GetObjectVersion"
            ],
            "Resource": "arn:aws:s3:::*"
        }
    ]
}

透過 AWS CLI 建立 Policy

$ aws iam create-policy --policy-name my-s3 --policy-document file://iam-policy.json

{
    "Policy": {
        "PolicyName": "my-s3",
        "PolicyId": "ANPAYFMQSNSE2LL6JJT3Q",
        "Arn": "arn:aws:iam::111111111111:policy/my-s3",
        "Path": "/",
        "DefaultVersionId": "v1",
        "AttachmentCount": 0,
        "PermissionsBoundaryUsageCount": 0,
        "IsAttachable": true,
        "CreateDate": "2022-09-23T10:56:00+00:00",
        "UpdateDate": "2022-09-23T10:56:00+00:00"
    }
}

透過 eksctl 命令建立 Kubernetes Service Account 及 IAM role。此會透過 CloudFormation 建立對應的 role。

$ eksctl create iamserviceaccount \
  --name my-s3-sa \
  --namespace default \
  --cluster ironman-2022 \
  --attach-policy-arn arn:aws:iam::111111111111:policy/my-s3 \
  --approve
2022-09-23 10:56:52 [ℹ]  1 existing iamserviceaccount(s) (kube-system/aws-node) will be excluded
2022-09-23 10:56:52 [ℹ]  1 iamserviceaccount (default/my-s3-sa) was included (based on the include/exclude rules)
2022-09-23 10:56:52 [!]  serviceaccounts that exist in Kubernetes will be excluded, use --override-existing-serviceaccounts to override
2022-09-23 10:56:52 [ℹ]  1 task: {
    2 sequential sub-tasks: {
        create IAM role for serviceaccount "default/my-s3-sa",
        create serviceaccount "default/my-s3-sa",
    } }2022-09-23 10:56:52 [ℹ]  building iamserviceaccount stack "eksctl-ironman-2022-addon-iamserviceaccount-default-my-s3-sa"
2022-09-23 10:56:52 [ℹ]  deploying stack "eksctl-ironman-2022-addon-iamserviceaccount-default-my-s3-sa"
2022-09-23 10:56:52 [ℹ]  waiting for CloudFormation stack "eksctl-ironman-2022-addon-iamserviceaccount-default-my-s3-sa"
2022-09-23 10:57:22 [ℹ]  waiting for CloudFormation stack "eksctl-ironman-2022-addon-iamserviceaccount-default-my-s3-sa"
2022-09-23 10:57:22 [ℹ]  created serviceaccount "default/my-s3-sa"

建立 Pod 使用 AWS CLI image 作為測試，儲存為 aws-cli.yaml YAML。

$ cat ./aws-cli.yaml
apiVersion: v1
kind: Pod
metadata:
  name: aws-cli
  namespace: default
spec:
  serviceAccountName: my-s3-sa
  containers:
  - name: aws-cli
    image: amazon/aws-cli:latest
    command:
      - sleep
      - infinity
    imagePullPolicy: IfNotPresent
  restartPolicy: Always

部署此 Pod。

$ kubectl apply -f ./aws-cli.yaml
pod/aws-cli created

分析

透過 kubectl describe 命令查看 Pod，以下列出部分資訊：

$ kubectl describe po aws-cli
Name:         aws-cli
Namespace:    default
...
...
Containers:
  aws-cli:
    ...
    Command:
      sleep
      infinity
    State:          Running
      Started:      Fri, 23 Sep 2022 10:59:13 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      AWS_STS_REGIONAL_ENDPOINTS:   regional
      AWS_DEFAULT_REGION:           eu-west-1
      AWS_REGION:                   eu-west-1
      AWS_ROLE_ARN:                 arn:aws:iam::111111111111:role/eksctl-ironman-2022-addon-iamserviceaccount-defau-Role1-KXGCZM9EIHAA
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cvtwf (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  aws-iam-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  86400
  kube-api-access-cvtwf:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
...

可以觀察多了一些設定：

環境變數 Environment： AWS_STS_REGIONAL_ENDPOINTS、AWS_DEFAULT_REGION、AWS_REGION、AWS_ROLE_ARN、AWS_WEB_IDENTITY_TOKEN_FILE
掛載目錄 Mount：
- /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
- /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cvtwf (ro)

根據 Day6 我們所探討過的 Mutating Webhook，是有可能在 Pod 部署過程中更新 Pod Spec。

filter @logStream like /^kube-apiserver-audit/
 | fields @timestamp, @message
 | sort @timestamp asc
 | filter objectRef.name == 'aws-cli' AND verb != 'get' AND verb != 'list'
 | limit 10000

從 audit logs 中，我們可以觀察到 mutating webhook pod-identity-webhook 更新了對應 annotation、environment variable、Mount 等資訊：

annotations.patch.webhook.admission.k8s.io/round_0_index_2: {"configuration":"pod-identity-webhook","webhook":"iam-for-pods.amazonaws.com","patch":[{"op":"add","path":"/spec/volumes/0","value":{"name":"aws-iam-token","projected":{"sources":[{"serviceAccountToken":{"audience":"sts.amazonaws.com","expirationSeconds":86400,"path":"token"}}]}}},{"op":"add","path":"/spec/containers","value":[{"name":"aws-cli","image":"amazon/aws-cli:latest","command":["sleep","infinity"],"env":[{"name":"AWS_STS_REGIONAL_ENDPOINTS","value":"regional"},{"name":"AWS_DEFAULT_REGION","value":"eu-west-1"},{"name":"AWS_REGION","value":"eu-west-1"},{"name":"AWS_ROLE_ARN","value":"arn:aws:iam::111111111111:role/eksctl-ironman-2022-addon-iamserviceaccount-defau-Role1-KXGCZM9EIHAA"},{"name":"AWS_WEB_IDENTITY_TOKEN_FILE","value":"/var/run/secrets/eks.amazonaws.com/serviceaccount/token"}],"resources":{},"volumeMounts":[{"name":"kube-api-access-cvtwf","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"},{"name":"aws-iam-token","readOnly":true,"mountPath":"/var/run/secrets/eks.amazonaws.com/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}]}],"patchType":"JSONPatch"}

近一步查看這個 Service Account Token 是什麼，其格式似乎是 JSON Web Tokens（JWT）² token 格式：

$ kubectl exec -it aws-cli -- cat /var/run/secrets/eks.amazonaws.com/serviceaccount/token
eyJhbGciOiJSUzI1NiIsImtpZCI6IjFlMWQzNzMyZGE0ZjY3OGQyOTYyOWFhODkwNDUzNWMzNTRhNTBjYzkifQ.eyJhdWQiOlsic3RzLmFtYXpvbmF3cy5jb20iXSwiZXhwIjoxNjY0MDE3MTQyLCJpYXQiOjE2NjM5MzA3NDIsImlzcyI6Imh0dHBzOi8vb2lkYy5la3MuZXUtd2VzdC0xLmFtYXpvbmF3cy5jb20vaWQvQThFN0EzOUNBRUJFRjZBQTkyNTBERkE5MzY2RkRGQTIiLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6ImRlZmF1bHQiLCJwb2QiOnsibmFtZSI6ImF3cy1jbGkiLCJ1aWQiOiJlNjIzY2U4Yi0zOGEyLTQxNmQtYTgzOS05ZmU3ZDMzYTA0ZWIifSwic2VydmljZWFjY291bnQiOnsibmFtZSI6Im15LXMzLXNhIiwidWlkIjoiMTFmNWFhMjAtMDIxYi00MmI5LWI0N2YtOWU0MDVlYTdkOTMyIn19LCJuYmYiOjE2NjM5MzA3NDIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0Om15LXMzLXNhIn0.kQMRT47oEs4hO62Zv61qFMllnxjt6stsqtRpaIPypZ69-TaasCsE-LzME1aEqh3AeGjJxCqVMWqq4_AP5z1lL_bwSpfB4TJDcezGCVf1cp-Ko4jQ61W5GvJzPxsg_u_xdXKXppFw5vsDHsR0xoGG-3z8D_9VJRcnTOEm9azT_l6LkdqRfuRcG9Vys-KbBAtI68DNAYDnn6P2voddltvaBytS8FgYfHrAapf0UwKN059MpH5icHbhJAvV45oXrZsVs6_gAaYVPZaSRateQ-Z6d6JZVb9KDJl7T1RvX77ers4l8-lLGLeA4ZpyIYbljckOSmi39O-UJ2lVw1dFJziMKg

透過上述 JWT 頁面 decode 後，我們可以關注 payload 資料部分，其中我們關注到 issuer (iss) 正是 EKS OIDC provider URL：

{
  "aud": [
    "sts.amazonaws.com"
  ],
  "exp": 1664017142,
  "iat": 1663930742,
  "iss": "https://oidc.eks.eu-west-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
  "kubernetes.io": {
    "namespace": "default",
    "pod": {
      "name": "aws-cli",
      "uid": "e623ce8b-38a2-416d-a839-9fe7d33a04eb"
    },
    "serviceaccount": {
      "name": "my-s3-sa",
      "uid": "11f5aa20-021b-42b9-b47f-9e405ea7d932"
    }
  },
  "nbf": 1663930742,
  "sub": "system:serviceaccount:default:my-s3-sa"
}

我們也可以透過 aws eks describe-cluster 命令來查看 OIDC provider URL：

$ aws eks describe-cluster --name=ironman-2022 --output=text --query 'cluster.identity.oidc'
https://oidc.eks.eu-west-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

一般來說，若要使用 IRSA 功能，首先第一步驟是建立 IAM OIDC Provider ³ 於 EKS cluster 上。然而此步驟於先前定義 eksctl ClusterConfig 時則設定：

iam:
  withOIDC: true

總結

由上述初步查看 Pod 及 audit log 可以了解：

pod-identity-webhook 會替我們注入對應環境變數及掛載 service account token
其中 token 格式為 JWT 並且由 EKS IAM OIDC endpoint 所發放

下一篇我們將會探討 pod-identity-webhook 如何實現掛載 Service Account Token 及 Kubernetes 為什麼可以整合 IAM 及此 token 用處為何。

參考文件

IAM roles for service accounts - https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html ↩
JWT - https://jwt.io/ ↩
Creating an IAM OIDC provider for your cluster - https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html ↩

[08] 為什麼 EKS 可以整合 IAM roles for service accounts（IRSA）（一）

建置環境​

分析​

總結​

參考文件​

Footnotes​

建置環境

分析

總結

參考文件

Footnotes