[23] 為什麼 EKS worker node IP address 容易佔用 IP 或是 subnets IP 地址不夠（一）
在 EKS 環境中整合 AWS VPC 環境，基本上 Pod 所使用的 IP 地址仍是使用 VPC subnets 內的 IP 地址，但是有時候我們觀察到 Node 實際上執行中的 Pod 並沒有那麼多，但是 VPC subnets 可用的 IP 地址數量經常被佔用。
因此本文將探討 EKS worker node 節點是如何分配 IP 地址於節點上。
檢視環境
在本系列文章中我們是藉由 eksctl 命令建立 EKS cluster 並未特別設定 VPC networking 相關設定而使用預設數值。以下查看預設 EKS VPC 及 subnets CIDR 範圍大小及當前 EKS 環境中所使用的 Node 及 Pod 資源狀況：
- 透過
aws eks describe-cluster命令查看 VPC 及 subnets 資訊。
$ aws eks describe-cluster --name ironman-2022 --query cluster.resourcesVpcConfig
{
"subnetIds": [
"subnet-0e863f9fbcda592a3",
"subnet-00ebeb2e8903fb3f9",
"subnet-02d98be342d8ab2a7",
"subnet-0282c441c5b9ce877",
"subnet-0b85d98c988f836ba",
"subnet-06bc5a0b5f0cb136b"
],
"securityGroupIds": [
"sg-0740a64625e50e048"
],
"clusterSecurityGroupId": "sg-032f6354cf0b25196",
"vpcId": "vpc-0b150640c72b722db",
"endpointPublicAccess": true,
"endpointPrivateAccess": false,
"publicAccessCidrs": [
"0.0.0.0/0"
]
}
- 分別透過
describe-vpcs及describe-subnets命令查看 VPC 及 subnets CIDR。
$ aws ec2 describe-vpcs --vpc-ids vpc-0b150640c72b722db --query "Vpcs[*].CidrBlock" --output=text
192.168.0.0/16
$ aws ec2 describe-subnets --subnet-ids subnet-0e863f9fbcda592a3 subnet-00ebeb2e8903fb3f9 subnet-02d98be342d8ab2a7 subnet-0282c441c5b9ce877 subnet-0b85d98c988f836ba subnet-06bc5a0b5f0cb136b --query "Subnets[*].CidrBlock"
[
"192.168.160.0/19",
"192.168.64.0/19",
"192.168.0.0/19",
"192.168.96.0/19",
"192.168.128.0/19",
"192.168.32.0/19"
]
- 目前 EKS cluster 僅存使用一組 managed node group
ng1-public-ssh，使用 instance typem5.large數量為 5。
$ eksctl get ng --cluster=ironman-2022
CLUSTER NODEGROUP STATUS CREATED MIN SIZE MAX SIZE DESIRED CAPACITY INSTANCE TYPE IMAGE ID ASG NAME TYPE
ironman-2022 ng1-public-ssh ACTIVE 2022-09-14T09:54:36Z 0 10 5 m5.large AL2_x86_64 eks-ng1-public-ssh-62c19db5-f965-bdb7-373a-147e04d9f124 managed
- 使用
kubectl命令檢視 node 及 pod 資源。
$ kubectl get no
NAME STATUS ROLES AGE VERSION
ip-192-168-18-190.eu-west-1.compute.internal Ready <none> 3d1h v1.22.12-eks-ba74326
ip-192-168-55-245.eu-west-1.compute.internal Ready <none> 3d1h v1.22.12-eks-ba74326
ip-192-168-61-79.eu-west-1.compute.internal Ready <none> 2d6h v1.22.12-eks-ba74326
ip-192-168-71-85.eu-west-1.compute.internal Ready <none> 2d6h v1.22.12-eks-ba74326
ip-192-168-76-241.eu-west-1.compute.internal Ready <none> 2d6h v1.22.12-eks-ba74326
$ kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system aws-node-5gg2g 1/1 Running 1 (2d6h ago) 2d6h
kube-system aws-node-hbb9k 1/1 Running 0 3d1h
kube-system aws-node-rbbsr 1/1 Running 1 (3d1h ago) 3d1h
kube-system aws-node-vhgmj 1/1 Running 0 2d6h
kube-system aws-node-x7vv5 1/1 Running 1 (2d6h ago) 2d6h
kube-system coredns-5947f47f5f-r4n48 1/1 Running 1 (3d1h ago) 3d1h
kube-system coredns-5947f47f5f-zb4cq 1/1 Running 0 2d6h
kube-system kube-proxy-4dpnm 1/1 Running 1 (3d1h ago) 3d1h
kube-system kube-proxy-btz4n 1/1 Running 0 2d6h
kube-system kube-proxy-j8hhz 1/1 Running 0 3d1h
kube-system kube-proxy-pbgzd 1/1 Running 1 (2d6h ago) 2d6h
kube-system kube-proxy-pm595 1/1 Running 1 (2d6h ago) 2d6h
nginx-demo nginx-demo-deployment-7489d44bbd-9z5dg 1/1 Running 0 2d3h
nginx-demo nginx-demo-deployment-7489d44bbd-jcshp 1/1 Running 0 2d3h
nginx-demo nginx-demo-deployment-7489d44bbd-m22pd 1/1 Running 0 2d3h
- 於 VPC 控制台可以查看剩餘可用 IP 地址。
- subnet-06bc5a0b5f0cb136b(192.168.160.0/19): 8186
- subnet-0282c441c5b9ce877(192.168.96.0/19): 8185
- subnet-0b85d98c988f836ba(192.168.128.0/19): 8185
- subnet-0e863f9fbcda592a3(192.168.0.0/19): 8166
- subnet-02d98be342d8ab2a7(192.168.64.0/19): 8157
- subnet-00ebeb2e8903fb3f9(192.168.32.0/19): 8147
由上述輸出資訊我們可以知道，每一個 subnet 使用 /19，基本上有 8190 個 IP 地址可以使用。而在 EKS 環境中僅在執行的 Pod 數量為 15 個，倘若進一步扣除使用 host network 的 aws-node 及 kube-proxy Pod，會關聯非主機 IP 地址的 Pod 僅剩 5 個 Pod，但是我們卻能觀察到如 subnet subnet-00ebeb2e8903fb3f9 僅剩 8147 IP 數量，使用 43 個 IP 地址了。
$ ipcalc 192.168.32.0/19
Address: 192.168.32.0 11000000.10101000.001 00000.00000000
Netmask: 255.255.224.0 = 19 11111111.11111111.111 00000.00000000
Wildcard: 0.0.31.255 00000000.00000000.000 11111.11111111
=>
Network: 192.168.32.0/19 11000000.10101000.001 00000.00000000
HostMin: 192.168.32.1 11000000.10101000.001 00000.00000001
HostMax: 192.168.63.254 11000000.10101000.001 11111.11111110
Broadcast: 192.168.63.255 11000000.10101000.001 11111.11111111
Hosts/Net: 8190 Class C, Private Internet
然而我們查看啟用於 subnet-00ebeb2e8903fb3f9 subnet 的節點數量有 i-03846c730abe852f4 及 i-020fc35362114539d 兩個 EC2 。透過以下 describe-instances 查看關聯的 interface 資訊：
$ aws ec2 describe-instances --instance-ids i-03846c730abe852f4 --query "Reservations[*].Instances[*].NetworkInterfaces[]"
[
[
{
"Attachment": {
"AttachTime": "2022-10-05T13:16:05+00:00",
"AttachmentId": "eni-attach-09a6558d224b5dcf4",
"DeleteOnTermination": true,
"DeviceIndex": 1,
"Status": "attached",
"NetworkCardIndex": 0
},
"Description": "aws-K8S-i-03846c730abe852f4",
"Groups": [
{
"GroupName": "eksctl-ironman-2022-nodegroup-ng1-public-ssh-remoteAccess",
"GroupId": "sg-079157a483bcb371f"
},
{
"GroupName": "eks-cluster-sg-ironman-2022-961501171",
"GroupId": "sg-032f6354cf0b25196"
}
],
"Ipv6Addresses": [],
"MacAddress": "0a:92:eb:9f:86:db",
"NetworkInterfaceId": "eni-0bbea279695112970",
"OwnerId": "111111111111",
"PrivateDnsName": "ip-192-168-50-176.eu-west-1.compute.internal",
"PrivateIpAddress": "192.168.50.176",
"PrivateIpAddresses": [
{
"Primary": true,
"PrivateDnsName": "ip-192-168-50-176.eu-west-1.compute.internal",
"PrivateIpAddress": "192.168.50.176"
},
{
"Primary": false,
"PrivateDnsName": "ip-192-168-53-76.eu-west-1.compute.internal",
"PrivateIpAddress": "192.168.53.76"
},
{
"Primary": false,
"PrivateDnsName": "ip-192-168-41-30.eu-west-1.compute.internal",
"PrivateIpAddress": "192.168.41.30"
},
{
"Primary": false,
"PrivateDnsName": "ip-192-168-47-240.eu-west-1.compute.internal",
"PrivateIpAddress": "192.168.47.240"
},
{
"Primary": false,
"PrivateDnsName": "ip-192-168-47-144.eu-west-1.compute.internal",
"PrivateIpAddress": "192.168.47.144"
},
{
"Primary": false,
"PrivateDnsName": "ip-192-168-61-98.eu-west-1.compute.internal",
"PrivateIpAddress": "192.168.61.98"
},
{
"Primary": false,
"PrivateDnsName": "ip-192-168-35-36.eu-west-1.compute.internal",
"PrivateIpAddress": "192.168.35.36"
},
{
"Primary": false,
"PrivateDnsName": "ip-192-168-61-68.eu-west-1.compute.internal",
"PrivateIpAddress": "192.168.61.68"
},
{
"Primary": false,
"PrivateDnsName": "ip-192-168-51-85.eu-west-1.compute.internal",
"PrivateIpAddress": "192.168.51.85"
},
{
"Primary": false,
"PrivateDnsName": "ip-192-168-52-54.eu-west-1.compute.internal",
"PrivateIpAddress": "192.168.52.54"
}
],
"SourceDestCheck": true,
"Status": "in-use",
"SubnetId": "subnet-00ebeb2e8903fb3f9",
"VpcId": "vpc-0b150640c72b722db",
"InterfaceType": "interface"
},
{
"Association": {
"IpOwnerId": "amazon",
"PublicDnsName": "ec2-52-210-231-138.eu-west-1.compute.amazonaws.com",
"PublicIp": "52.210.231.138"
},
"Attachment": {
"AttachTime": "2022-10-05T13:14:38+00:00",
"AttachmentId": "eni-attach-011fd1ecaadbba547",
"DeleteOnTermination": true,
"DeviceIndex": 0,
"Status": "attached",
"NetworkCardIndex": 0
},
"Description": "",
"Groups": [
{
"GroupName": "eksctl-ironman-2022-nodegroup-ng1-public-ssh-remoteAccess",
"GroupId": "sg-079157a483bcb371f"
},
{
"GroupName": "eks-cluster-sg-ironman-2022-961501171",
"GroupId": "sg-032f6354cf0b25196"
}
],
"Ipv6Addresses": [],
"MacAddress": "0a:39:dc:28:ad:03",
"NetworkInterfaceId": "eni-0ab6702642861e690",
"OwnerId": "111111111111",
"PrivateDnsName": "ip-192-168-55-245.eu-west-1.compute.internal",
"PrivateIpAddress": "192.168.55.245",
"PrivateIpAddresses": [
{
"Association": {
"IpOwnerId": "amazon",
"PublicDnsName": "ec2-52-210-231-138.eu-west-1.compute.amazonaws.com",
"PublicIp": "52.210.231.138"
},
"Primary": true,
"PrivateDnsName": "ip-192-168-55-245.eu-west-1.compute.internal",
"PrivateIpAddress": "192.168.55.245"
},
{
"Primary": false,
"PrivateDnsName": "ip-192-168-49-56.eu-west-1.compute.internal",
"PrivateIpAddress": "192.168.49.56"
},
{
"Primary": false,
"PrivateDnsName": "ip-192-168-44-157.eu-west-1.compute.internal",
"PrivateIpAddress": "192.168.44.157"
},
{
"Primary": false,
"PrivateDnsName": "ip-192-168-38-15.eu-west-1.compute.internal",
"PrivateIpAddress": "192.168.38.15"
},
{
"Primary": false,
"PrivateDnsName": "ip-192-168-41-146.eu-west-1.compute.internal",
"PrivateIpAddress": "192.168.41.146"
},
{
"Primary": false,
"PrivateDnsName": "ip-192-168-60-179.eu-west-1.compute.internal",
"PrivateIpAddress": "192.168.60.179"
},
{
"Primary": false,
"PrivateDnsName": "ip-192-168-33-52.eu-west-1.compute.internal",
"PrivateIpAddress": "192.168.33.52"
},
{
"Primary": false,
"PrivateDnsName": "ip-192-168-49-180.eu-west-1.compute.internal",
"PrivateIpAddress": "192.168.49.180"
},
{
"Primary": false,
"PrivateDnsName": "ip-192-168-41-36.eu-west-1.compute.internal",
"PrivateIpAddress": "192.168.41.36"
},
{
"Primary": false,
"PrivateDnsName": "ip-192-168-42-86.eu-west-1.compute.internal",
"PrivateIpAddress": "192.168.42.86"
}
],
"SourceDestCheck": true,
"Status": "in-use",
"SubnetId": "subnet-00ebeb2e8903fb3f9",
"VpcId": "vpc-0b150640c72b722db",
"InterfaceType": "interface"
}
]
]
總結
由上述資訊，我們初步可以知道 EKS VPC 中子網大量消耗 IP 地址是由 EKS worker node 啟用時會被指派，明天我們將繼續探討為什麼 EKS worker node 預設會關聯使用這麼多的 IP 地址。