Having trouble configuring static IPs to PODs attached with MACVLAN interface

Question

Here is the scenario. There is a deployment set through which 2 PODs are created. I am attaching a MACVLAN interface to these PODs for external communication.

Macvlan definition

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: test-macvlandef01
spec:
  config: '{
      "cniVersion": "0.3.0",
      "name": "test-macvlandef01",
      "type": "macvlan",
      "master": "eth0",
      "mode": "bridge",
      "ipam": {
        "type": "whereabouts",
        "datastore": "kubernetes",
        "kubernetes": { "kubeconfig": "/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig" },
        "range": "192.168.0.0/24",
        "range_start": "192.168.0.44",
        "range_end": "192.168.0.45"
      }
    }'

Deployment Set

apiVersion: apps/v1
kind: Deployment
metadata:
  name: centos-test
spec:
  replicas: 2
  selector:
    matchLabels:
      app: centos
  template:
    metadata:
      labels:
        app: centos
      annotations:
       k8s.v1.cni.cncf.io/networks: "test-macvlandef01"
    spec:
      nodeSelector:
        test: "true"
      containers:
      - name: centos
        image: centos
        imagePullPolicy: IfNotPresent
        command: ["bin/bash", "-c", "sleep 100000" ]
        ports:
        - containerPort: 80

Result. Both PODs have IPs from the allocated pool.

[master1 ~]# kubectl exec -it centos-test-64f8fbf47f-wrjr7  ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if61: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default 
    link/ether 72:ef:ca:2c:31:3e brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.20.14.176/32 scope global eth0
       valid_lft forever preferred_lft forever
5: net1@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default 
    link/ether 52:2f:bd:f9:03:09 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.0.44/24 brd 192.168.0.255 scope global net1
       valid_lft forever preferred_lft forever
       
[master1 ~]# kubectl exec -it centos-test-64f8fbf47f-vtkst  ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if60: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default 
    link/ether ae:e6:4e:95:2a:f2 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.20.14.175/32 scope global eth0
       valid_lft forever preferred_lft forever
5: net1@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default 
    link/ether 72:fb:b5:90:d0:37 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.0.45/24 brd 192.168.0.255 scope global net1
       valid_lft forever preferred_lft forever

Now what I need to configure is, a bigger allocation pool in macvlan definition file, but have only specific 2 IPs to be assigned to the PODs. I tried below configuration.

Macvlan definition

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: test-macvlandef01
spec:
  config: '{
      "cniVersion": "0.3.0",
      "name": "test-macvlandef01",
      "type": "macvlan",
      "master": "eth0",
      "mode": "bridge",
      "ipam": {
        "type": "whereabouts",
        "datastore": "kubernetes",
        "kubernetes": { "kubeconfig": "/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig" },
        "range": "192.168.0.0/24",
        "range_start": "192.168.0.40",
        "range_end": "192.168.0.50"
      }
    }'

Deployment Set

apiVersion: apps/v1
kind: Deployment
metadata:
  name: centos-test
spec:
  replicas: 2
  selector:
    matchLabels:
      app: centos
  template:
    metadata:
      labels:
        app: centos
      annotations:
       k8s.v1.cni.cncf.io/networks: '[{ "name": "test-macvlandef01","ips": "192.168.0.44"},{"name": "test-macvlandef01","ips": "192.168.0.45"}]'
    spec:
      nodeSelector:
        test: "true"
      containers:
      - name: centos
        image: centos
        imagePullPolicy: IfNotPresent
        command: ["bin/bash", "-c", "sleep 100000" ]
        ports:
        - containerPort: 80

PODs are coming up without MACVLAN interface and also I see no error associated with the POD.

[master1 ~]# kubectl exec -it centos-test-b59db89f7-2vvqx  ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if65: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default 
    link/ether 62:31:fc:64:8f:5b brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.20.14.180/32 scope global eth0
       valid_lft forever preferred_lft forever

[master1 ~]# kubectl exec -it centos-test-b59db89f7-6c75h  ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if64: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default 
    link/ether e6:23:30:ff:bf:c3 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.20.14.179/32 scope global eth0
       valid_lft forever preferred_lft forever

Please suggest any modifications or additions that would help with the requirement.

Thanks in advance.

Answer 1

I want to pay your attention into 2 points below. Partial answer.

---

1. From your post I see that you want to use special IP addresses. To use such functionality, according to Extention convention from CNI you may need to use "capabilities": {"ips": true} capability in your Macvlan definition. Something like this:

spec:
  config: '{
      "cniVersion": "0.3.0",
      "name": "test-macvlandef01",
      "type": "macvlan",
      "capabilities": {"ips": true}
      "master": "eth0",
      "mode": "bridge",

You can also find good explanation with examples in Attaching a pod to an additional network documentation.

---

2. I suppose that you use whereabouts plugin, since "type": "whereabouts" presents in your Macvlan definition. It supports exclusions:

You can also specify ranges to exclude from assignment, so if for example you'd like to assign IP addresses within the range 192.168.2.0/24, you can exclude IP addresses within it by adding them to an exclude list. For example, if you decide to exclude the range 192.168.2.0/28, the first IP address assigned in the range will be 192.168.2.16.

Knowing this fact, you can specify ranges of IPs to exclude from your configuration in accordance with Whereabouts IPAM Config example. Try to add exclude field in Macvlan definition with necessary IPs/subnets, which should be excluded. Possible solution for your particular case:

spec:
  config: '{
      "cniVersion": "0.3.0",
      "name": "test-macvlandef01",
      "type": "macvlan",
      "capabilities": {"ips": true}
      "master": "eth0",
      "mode": "bridge",
      "ipam": {
        "type": "whereabouts",
        "range": "192.168.0.0/24",
        "range_start": "192.168.0.40",
        "range_end": "192.168.0.50"
        "exclude": [
           "192.168.0.40/32",
           "192.168.0.41/32",
           ...
        ]
      }
  }'