Terraform cyclic dependency issue when referencing IP addresses to generate config file

Question

I am trying to setup an AWS environment with 2 ec2 instances in a VPC that are configured to run a piece of software that requires a config file containing the IP address of the other ec2. To do this, I am creating the config file in a template that I am running to start the ec2 like this:

data "template_file" "init_relay" {
  template = file("${path.module}/initRelay.tpl")
  vars = {
    port    = var.node_communication_port
    ip      = module.block-producing-node.private_ip[0]
    self_ip = module.relay-node.public_ip
  }
}

module "relay-node" {
  source                      = "terraform-aws-modules/ec2-instance/aws"
  name                        = "relay-node"
  ami                         = var.node_ami
  key_name                    = "aws-keys"
  user_data                   = data.template_file.init_relay.rendered
  instance_type               = var.instance_type
  subnet_id                   = module.vpc.public_subnets[0]
  vpc_security_group_ids      = [module.relay_node_sg.this_security_group_id]
  associate_public_ip_address = true
  monitoring                  = true
  root_block_device = [
    {
      volume_type = "gp2"
      volume_size = 35
    },
  ]
  tags = {
    Name        = "Relay Node"
    Environment = var.environment_tag
    Version     = var.pool_version
  }
}

data "template_file" "init_block_producer" {
  template = "${file("${path.module}/initBlockProducer.tpl")}"
  vars = {
    port = var.node_communication_port
    ip = module.relay-node.private_ip
    self_ip       = module.block-producing-node.private_ip
  }
}

module "block-producing-node" {
  source                      = "terraform-aws-modules/ec2-instance/aws"
  name                        = "block-producing-node"
  ami                         = var.node_ami
  key_name                    = "aws-keys"
  user_data                   = data.template_file.init_block_producer.rendered
  instance_type               = var.instance_type
  subnet_id                   = module.vpc.public_subnets[0]
  vpc_security_group_ids      = [module.block_producing_node_sg.this_security_group_id]
  associate_public_ip_address = true
  monitoring                  = true
  root_block_device = [
    {
      volume_type = "gp2"
      volume_size = 35
    },
  ]
  tags = {
    Name        = "Block Producing Node"
    Environment = var.environment_tag
    Version     = var.pool_version
  }
}

but that gives me a cyclic dependency error:

» terraform apply

Error: Cycle: module.relay-node.output.public_ip, module.block-producing-node.output.private_ip, data.template_file.init_relay, module.relay-node.var.user_data, module.relay-node.aws_instance.this, module.relay-node.output.private_ip, data.template_file.init_block_producer, module.block-producing-node.var.user_data, module.block-producing-node.aws_instance.this

To me that makes sense why I am getting this error because in order to generate the config file for one ec2, the other ec2 already needs to exist and have a ip address assigned to it. But I don't know how to do this in a way.

How do I reference the IP address of the other EC2 in the template file in a way that doesn't cause a cyclic dependency issue?

Answer 1

Generally-speaking, the user data of an EC2 instance cannot contain any of the IP addresses of the instance because the user data is submitted as part of launching the instance and cannot be changed after the instance is launched, and the IP address (unless you specify an explicit one when launching) is also assigned during instance launch, as part of creating the implied main network interface.

If you have only a single instance and it needs to know its own IP address then the easiest answer is for some software installed in your instance to ask the operating system which IP address has been assigned to the main network interface. The operating system already knows the IP address as part of configuring the interface using DHCP, and so there's no need to also pass it in via user data.

A more common problem, though, is when you have a set of instances that all need to talk to each other, such as to form some sort of cluster, and so they need the IP addresses of their fellows in addition to their own IP addresses. In that situation, there are broadly-speaking two approaches:

• Arrange for Terraform to publish the IP addresses somewhere that will allow the software running in the instances to retrieve them after the instance has booted.

  For example, you could publish the list in AWS SSM Parameter Store using aws_ssm_parameter and then have the software in your instance retrieve it from there, or you could assign all of your instances into a VPC security group and then have the software in your instance query the VPC API to enumerate the IP addresses of all of the network interfaces that belong to that security group.

  All variants of this strategy have the problem that the software in your instances may start up before the IP address data is available or before it's complete. Therefore it's usually necessary to periodically poll whatever data source is providing the IP addresses in case new addresses appear. On the other hand, that capability also lends itself well to autoscaling systems where Terraform is not directly managing the instances.

  This is the technique used by ElasticSearch EC2 Discovery, for example, looking for network interfaces belonging to a particular security group, or carrying specific tags, etc.

• Reserve IP addresses for your instances ahead of creating them so that the addresses will be known before the instance is created.

  When we create an aws_instance without saying anything about network interfaces, the EC2 system implicitly creates a primary network interface and chooses a free IP address from whatever subnet the instance is bound to. However, you have the option to create your own network interfaces that are managed separately from the instances they are attached to, which both allows you to reserve a private IP address without creating an instance and allows a particular network interface to be detached from one instance and then connected to another, preserving the reserved IP address.

  aws_network_interface is the AWS provider resource type for creating an independently-managed network interface. For example:

  resource "aws_network_interface" "example" {
    subnet_id = aws_subnet.example.id
  }    
  

  The aws_network_interface resource type has a private_ips attribute whose first element is equivalent to the private_ip attribute on an aws_instance, so you can refer to aws_network_interface.example.private_ips[0] to get the IP address that was assigned to the network interface when it was created, even though it's not yet attached to any EC2 instance.

  When you declare the aws_instance you can include a network_interface block to ask EC2 to attach the pre-existing network interface instead of creating a new one:

  resource "aws_instance" "example" {
    # ...
  
    user_data = templatefile("${path.module}/user_data.tmpl", {
      private_ip = aws_network_interface.example.private_ips[0]
    })
  
    network_interface {
      device_index         = 0 # primary interface
      network_interface_id = aws_network_interface.example.id
    }
  }
  

  Because the network interface is now a separate resource, you can use its attributes as part of the instance configuration. I showed only a single network interface and a single instance above in order to focus on the question as stated, but you could also use resource for_each or count on both resources to create a set of instances and then use aws_network_interface.example[*].private_ips[0] to pass all of the IP addresses into your user_data template.

  A caveat with this approach is that because the network interfaces and instances are separate it is likely that a future change will cause an instance to be replaced without also replacing its associated network interface. That will mean that a new instance will be assigned the same IP address as an old one that was already a member of the cluster, which may be confusing to a system that uses IP addresses to uniquely identify cluster members. Whether that is important and what you'd need to do to accommodate it will depend on what software you are using to form the cluster.

  This approach is also not really suitable for use with an autoscaling system, because it requires the number of assigned IP addresses to grow and shrink in accordance with the current number of instances, and for the existing instances to somehow become aware when another instance joins or leaves the cluster.

Answer 2

Your template is dependent on your module and your module on your template - that is causing the cycle.

ip  = module.block-producing-node.private_ip[0]

and

user_data = data.template_file.init_block_producer.rendered