[14] 為什麼 security group 可以關聯至單獨的 EKS Pod - Security groups for pods（一）
EKS 使用 VPC 作為網路環境，在預設情況之下，Pod 僅能繼承 EKS worker node 上的 security group 限縮 inbound/outbound 流量。security group1 類似於 AWS VPC 環境中的防火牆概念，可以針對 IP CIDR 及 port 設置規則。而在 EKS VPC CNI Plugin 延伸 VPC security group 功能使 Pods 可以關聯指定 security group - Security groups for pods 2。
建置環境
- 確認當前 VPC CNI plugin 版本
v1.10.1。
$ kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2
amazon-k8s-cni-init:v1.10.1-eksbuild.1
amazon-k8s-cni:v1.10.1-eksbuild.1
- 查看 cluster 所使用的 EKS cluster IAM role3。
$ aws eks describe-cluster --name ironman-2022 --query cluster.roleArn --output text
arn:aws:iam::111111111111:role/eksctl-ironman-2022-cluster-ServiceRole-33QTRI4AJ85B
- 將 Policy
AmazonEKSVPCResourceController關聯至 EKS cluster IAM role。
aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/AmazonEKSVPCResourceController --role-name eksctl-ironman-2022-cluster-ServiceRole-33QTRI4AJ85B
- 設定
aws-node環境變數ENABLE_POD_ENI為true。
$ kubectl set env daemonset aws-node -n kube-system ENABLE_POD_ENI=true
daemonset.apps/aws-node env updated
- 確認
aws-nodePod 更新。更新後，可以觀察到 node 更新 labelvpc.amazonaws.com/has-trunk-attached。
$ kubectl -n kube-system get po
NAME READY STATUS RESTARTS AGE
aws-node-5mhsb 1/1 Running 0 2m
aws-node-fdmrf 0/1 Init:0/1 0 13s
aws-node-st44l 0/1 PodInitializing 0 13s
aws-node-vsqr4 0/1 Init:0/1 0 12s
...
$ kubectl get nodes -l vpc.amazonaws.com/has-trunk-attached=true
NAME STATUS ROLES AGE VERSION
ip-192-168-3-47.eu-west-1.compute.internal Ready <none> 3d15h v1.22.12-eks-ba74326
ip-192-168-31-166.eu-west-1.compute.internal Ready <none> 111m v1.22.12-eks-ba74326
ip-192-168-42-19.eu-west-1.compute.internal Ready <none> 111m v1.22.12-eks-ba74326
ip-192-168-71-180.eu-west-1.compute.internal Ready <none> 111m v1.22.12-eks-ba74326
- 分別建立以下
SecurityGroupPolicy（sg-policy.yaml）及測試aws-cliPod （aws-cli.yaml）。
$ cat ./sg-policy.yaml
apiVersion: vpcresources.k8s.aws/v1beta1
kind: SecurityGroupPolicy
metadata:
name: my-security-group-policy-demo
namespace: default
spec:
podSelector:
matchLabels:
role: demo-sg
securityGroups:
groupIds:
- sg-079157a483bcb371f
$ cat ./aws-cli.yaml
apiVersion: v1
kind: Pod
metadata:
name: aws-cli
namespace: default
labels:
role: demo-sg
spec:
containers:
- name: aws-cli
image: amazon/aws-cli:latest
command:
- sleep
- infinity
imagePullPolicy: IfNotPresent
restartPolicy: Always
- 部署
SecurityGroupPolicy及Pod。
$ kubectl apply -f ./sg-policy.yaml
securitygrouppolicy.vpcresources.k8s.aws/my-security-group-policy-demo created
$ kubectl apply -f ./aws-cli.yaml
pod/aws-cli created
分析
Pod
透過 kubectl describe 命令查看 Pod，以下列出部分資訊：
$ kubectl describe po aws-cli
Name: aws-cli
Namespace: default
Priority: 0
Node: ip-192-168-31-166.eu-west-1.compute.internal/192.168.31.166
Start Time: Thu, 29 Sep 2022 12:25:43 +0000
Labels: role=demo-sg
Annotations: kubernetes.io/psp: eks.privileged
vpc.amazonaws.com/pod-eni:
[{"eniId":"eni-09a7a909e06fc4be6","ifAddress":"02:b0:01:6c:b5:79","privateIp":"192.168.26.162","vlanId":1,"subnetCidr":"192.168.0.0/19"}]
Status: Running
IP: 192.168.26.162
IPs:
IP: 192.168.26.162
Containers:
aws-cli:
Container ID:
...
...
...
Restart Count: 0
Limits:
vpc.amazonaws.com/pod-eni: 1
Requests:
vpc.amazonaws.com/pod-eni: 1
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ldh4s (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-ldh4s:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
vpc.amazonaws.com/pod-eni:NoSchedule op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 74s default-scheduler Successfully assigned default/aws-cli to ip-192-168-31-166.eu-west-1.compute.internal
Normal SecurityGroupRequested 74s vpc-resource-controller Pod will get the following Security Groups [sg-079157a483bcb371f]
Normal ResourceAllocated 73s vpc-resource-controller Allocated [{"eniId":"eni-09a7a909e06fc4be6","ifAddress":"02:b0:01:6c:b5:79","privateIp":"192.168.26.162","vlanId":1,"subnetCidr":"192.168.0.0/19"}] to the pod
Warning FailedCreatePodSandBox 73s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e752a83b0e67ab7c60e6f3ec294c4cf6b7451802682d76a8addafa40f341fd7a" network for pod "aws-cli": networkPlugin cni failed to set up pod "aws-cli_default" network: add cmd: failed to assign an IP address to container
Normal SandboxChanged 72s kubelet Pod sandbox changed, it will be killed and re-created.
Normal Pulling 72s kubelet Pulling image "amazon/aws-cli:latest"
Normal Pulled 62s kubelet Successfully pulled image "amazon/aws-cli:latest" in 9.897131693s
Normal Created 60s kubelet Created container aws-cli
Normal Started 60s kubelet Started container aws-cli
可以觀察到多了 vpc.amazonaws.com/pod-eni annotation 資訊： [{"eniId":"eni-09a7a909e06fc4be6","ifAddress":"02:b0:01:6c:b5:79","privateIp":"192.168.26.162","vlanId":1,"subnetCidr":"192.168.0.0/19"}]
使用 CloudWatch Logs insights 以下語法查看 audit logs：
filter @logStream like /^kube-apiserver-audit/
| fields @timestamp, @message
| sort @timestamp asc
| filter objectRef.name == 'aws-cli' AND verb == 'create'
| limit 10000
我們可以觀察到由 mutating wenhook vpc-resource-mutating-webhook 更新此 annotation。
...
...
"annotations": {
"authorization.k8s.io/decision": "allow",
"authorization.k8s.io/reason": "",
"mutation.webhook.admission.k8s.io/round_0_index_0": "{\"configuration\":\"0500-amazon-eks-fargate-mutation.amazonaws.com\",\"webhook\":\"0500-amazon-eks-fargate-mutation.amazonaws.com\",\"mutated\":false}",
"mutation.webhook.admission.k8s.io/round_0_index_2": "{\"configuration\":\"pod-identity-webhook\",\"webhook\":\"iam-for-pods.amazonaws.com\",\"mutated\":false}",
"mutation.webhook.admission.k8s.io/round_0_index_3": "{\"configuration\":\"vpc-resource-mutating-webhook\",\"webhook\":\"mpod.vpc.k8s.aws\",\"mutated\":true}",
"mutation.webhook.admission.k8s.io/round_1_index_2": "{\"configuration\":\"pod-identity-webhook\",\"webhook\":\"iam-for-pods.amazonaws.com\",\"mutated\":false}",
"mutation.webhook.admission.k8s.io/round_1_index_3": "{\"configuration\":\"vpc-resource-mutating-webhook\",\"webhook\":\"mpod.vpc.k8s.aws\",\"mutated\":false}",
"patch.webhook.admission.k8s.io/round_0_index_3": "{\"configuration\":\"vpc-resource-mutating-webhook\",\"webhook\":\"mpod.vpc.k8s.aws\",\"patch\":[{\"op\":\"add\",\"path\":\"/spec/containers/0/resources/limits\",\"value\":{\"vpc.amazonaws.com/pod-eni\":\"1\"}},{\"op\":\"add\",\"path\":\"/spec/containers/0/resources/requests\",\"value\":{\"vpc.amazonaws.com/pod-eni\":\"1\"}}],\"patchType\":\"JSONPatch\"}",
"podsecuritypolicy.policy.k8s.io/admit-policy": "eks.privileged",
"podsecuritypolicy.policy.k8s.io/validate-policy": "eks.privileged"
}
...
Node
根據文件步驟，我們可以透過 Node 檢視 vpc.amazonaws.com/has-trunk-attached label 來確認環境變數 ENABLE_POD_ENI 是否設定成功。因此使用 kubectl describe 查看對應的更新：
$ kubectl describe no ip-192-168-31-166.eu-west-1.compute.internal
Name: ip-192-168-31-166.eu-west-1.compute.internal
Roles: <none>
Labels: alpha.eksctl.io/cluster-name=ironman-2022
alpha.eksctl.io/nodegroup-name=ng1-public-ssh
...
...
...
vpc.amazonaws.com/has-trunk-attached=true
Annotations: node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
...
Capacity:
...
...
vpc.amazonaws.com/pod-eni: 9
Allocatable:
...
...
vpc.amazonaws.com/pod-eni: 9
...
...
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 125m (6%) 0 (0%)
memory 0 (0%) 0 (0%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
attachable-volumes-aws-ebs 0 0
vpc.amazonaws.com/pod-eni 1 1
如同 Pod 一樣，可以觀察到多了 vpc.amazonaws.com/pod-eni annotation 資訊：
- label
vpc.amazonaws.com/has-trunk-attached=true - Capacity 數量為 9
- Allocated resources 數量為 1
接續，透過 CloudWatch Logs insights 以下語法查看 audit logs：
filter @logStream like /^kube-apiserver-audit/
| fields objectRef.name, objectRef.resource, @message
| sort @timestamp asc
| filter @message like 'vpc-resource-controller' AND verb == 'patch' AND objectRef.name == 'ip-192-168-31-166.eu-west-1.compute.internal'
| limit 10000
...
"verb": "patch",
"user": {
"username": "system:serviceaccount:kube-system:eks-vpc-resource-controller",
"uid": "b75dc772-f4fe-44ef-b928-5a1ba8a18eb4",
"groups": [
"system:serviceaccounts",
"system:serviceaccounts:kube-system",
"system:authenticated"
]
},
"sourceIPs": [
"10.0.51.57"
],
"userAgent": "controller/v0.0.0 (linux/amd64) kubernetes/$Format",
"objectRef": {
"resource": "nodes",
"name": "ip-192-168-31-166.eu-west-1.compute.internal",
"apiVersion": "v1",
"subresource": "status"
},
"responseStatus": {
"metadata": {},
"code": 200
},
"requestObject": {
"status": {
"capacity": {
"vpc.amazonaws.com/pod-eni": "9"
}
}
},
...
可以確認此 capacity 是由 username eks-vpc-resource-controller 更新此 node capacity 資訊 vpc.amazonaws.com/pod-eni。
總結
由上述初步查看 Node、Pod 資訊及 audit log 可以了解：
- 啟用 VPC CNI plugin
ENABLE_POD_ENI環境變數後，eks-vpc-resource-controller會更新 Node label 及vpc.amazonaws.com/pod-eni資訊。 - Mutating Webhook
vpc-resource-mutating-webhook於 Pod mutating 階段更新vpc.amazonaws.com/pod-eni資訊於 annotation。
下一篇我們將會探討如何實現 Pod 獨立於 Node 上的 security group。
參考文件
Footnotes
-
Control traffic to resources using security groups - https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html ↩
-
Security groups for pods - https://docs.aws.amazon.com/eks/latest/userguide/security-groups-for-pods.html ↩
-
Amazon EKS cluster IAM role - https://docs.aws.amazon.com/eks/latest/userguide/service_IAM_role.html ↩