Azure Load Balancer provisioning using Kubernetes Azure service operator

Question

I am trying to provision an Azure load balancer using the Azure service operator for Kubernetes. The load balancer Frontend IP configuration seems to be working just fine. What I am having trouble with is adding the virtual machine to the backend pool using its Ip address. Below is a snippet of my configuration:

apiVersion: network.azure.com/v1api20201101
kind: LoadBalancer
metadata:
  name: sampleloadbalancer
  namespace: default
spec:
  location: westcentralus
  owner:
    name: aso-sample-rg
  sku:
    name: Standard
  frontendIPConfigurations:
    - name: LoadBalancerFrontend
      privateIPAddress: 192.168.1.100
      privateIPAllocationMethod: "Static"
  backendAddressPools:
    - name: BackendPool
      loadBalancerBackendAddresses:
        - name: server-01
          ipAddress: 192.168.1.102
          subnet:
            reference:
              armId: /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/network-rg/providers/Microsoft.Network/virtualNetworks/myvnet/subnets/my-subnet-name
          virtualNetwork:
            reference:
              armId: /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/network-rg/providers/Microsoft.Network/virtualNetworks/myvnet
          loadBalancerFrontendIPConfiguration:
            reference:
              armId: /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/aso-sample-rg/providers/Microsoft.Network/loadBalancers/sampleloadbalancer/frontendIPConfigurations/LoadBalancerFrontend

Is what I am trying to do achievable? I have tried various combinations and went through the documentation and didn't manage to get the VM to be added to the pool.