Terraform tips and tricks

Essential links:

• Terraform
• Terraform interpolations
• Terraform backends
• Terraform modules
• Best practises

Chomp newlines

The chomp interpolation chomps the newline from a file. Useful if you store secrets in a filesystem (e.g. a different access-protected git repo):

db_password = "${chomp(file("/tf-secrets/mysql/db_password"))}"

Before chomp (pre-0.9.4), you would have to remove the newline from the file, or else your variable would be {file_contents}\n and cause problems.

Modules

There’s a lot to cover w.r.t. Terraform Modules, but a starting point is how I structure my TF repo:

/repos/tf $ tree -L 1
├── terraform
│   ├── consul-1-stg
│   └── vertica-v2-prd
└── terraform-modules
    ├── consul
    ├── public_vpc
    └── vertica

terraform contains provisioned resources. terraform-modules contains re-usable modules. You can (should?) have your modules in a separate git repo as well.

For now we’ll focus on a mono-repo approach to Terraform. From terraform, you can use modules with relative path imports:

module "vertica-v2-prd" {
  source            = "../../terraform-modules/vertica"

All of my variables are in the terraform/vertica-v2-prd directory, since this represents an instantiation of a Vertica cluster, with specific parameters (VPC, cluster size, etc.) that correspond to the logical stack (v2-prd).

The vertica module has no variables. Every variable it uses should come from the instantiation. Write this one as if you are sharing it with other people. Anything you hardcode will probably have to be painfully refactored at some point.

Variables

Continuing the above example, there are 3 variable files overall:

• terraform/vertica-v2-prd/variables.tf - defines blank variables that the stack needs
• terraform/vertica-v2-prd/terraform.tfvars - values for the above blank variables. If something is missing here, it will be prompted for interactively
• terraform-modules/vertica/variables.tf - defines blank variables that the stack needs

State mv

When refactoring Terraform from monolithic to modular, or changing module names, the terraform state mv command is incredibly important. It lets you move resources to their new name - this may seem simple but Terraform can’t automatically track resources across module renames.

Here’s an example command to take a separate route53-zone resource (that has been instantiated, and contains a terraform.tfstate file) and absorb it as a module inside a larger stack, with the module name my_custom_route53_zone:

terraform state mv -state=./route53-zone/terraform.tfstate -state-out=./terraform.tfstate aws_route53_record.primary-alias module.my_custom_route53_zone.aws_route53_record.primary-alias

Terraform automatically backs up your tfstate files when executing this command - be careful.

Using existing infrastructure

With backends, you can include outputs from other resources:

data "terraform_remote_state" "global_route53_zone" {
  backend = "local"

  config {
    path = "${path.root}/../global-route53/terraform.tfstate"
  }
}
...
module "my_stack" {
...
zone_id           = "${data.terraform_remote_state.global_route53_zone.zone_id}"
domain            = "${data.terraform_remote_state.global_route53_zone.domain}"
...
}

You can go one step further and use a remote backend (Artifactory, Consul, S3, etc.). This is good because it keeps your tfstate files separate from your code.

Null resource

Here's a way to execute things without consequences:

resource "null_resource" "provision_vertica_1" {
  connection {
    type        = "ssh"
    user        = "dbadmin"
    private_key = "${var.provision_keypair}"
    host        = "${aws_instance.vertica_node.0.public_ip}"
  }

  provisioner "remote-exec" {
    inline = [
      "sudo /opt/vertica/sbin/install_vertica --hosts ${join(",", aws_instance.vertica_node.*.private_ip)} --dba-user-password-disabled --point-to-point --ssh-identity /home/dbadmin/.ssh/vertica_install_key --accept-eula",
    ]
  }
}

Previously I attempted to have the provisioner block inside the aws_instance block defining my Vertica instances. However, there’s a self-referential problem there: the provision command needs the aws_instance.*.private_ip values as parameters.

null_resource solves that by allowing you to depend on the Vertica aws_instance existing.

Conditional actions

$ cat whatever.tf
resource "aws_instance" "optional_machine" {
  count = "${var.with_optional}"
  ...
}
$ cat variables.tf
variable "with_optional" { description = "Create optional instance" }
$ terraform plan
var.with_optional
  Create optional instance

  Enter a value:

Enter 0 or 1 (or more than 1 if you want) to toggle the creation of this instance.

Access a list index with formatlist

The formatlist interpolation formats all the elements of a list with the given string:

formatlist("%v:2888:3888", aws_instance.zookeeper.*.private_ip)

There is no special variable to access the list index (so you can print 0 from aws_instance.zookeeper.0), however it does allow you to format two lists of the same length. This is the solution.

• Add a tag

  tags {
   +    NodeId = "${count.index+1}"
      }
  

• Formatlist with the tag list

  formatlist("server.%v=%v:2888:3888", aws_instance.zookeeper.*.tags.NodeId, aws_instance.zookeeper.*.private_ip))}
  

Solution borrowed and adapted from this OpenStack example.

tfenv

tfenv is a great tool for versioning your Terraform repo.

I tried to spin my own but discovered tfenv is solid and use it (and recommend the rest of my team to use it). All of our Terraform HCL dirs have a .terraform-version file inside them.

Usage:

• Write a .terraform-version file, e.g. latest:^0.10
• Run tfenv install
• Run terraform as usual

Using tags in your modules repo

Without tags in your modules repo, something like this can be dangerous:

module "vertica_prod_nodes" {
  source            = "[email protected]:myorg/terraform-modules.git//vertica/nodes"

This is pointing to the tip of master in the myorg/terraform-modules repository and things can break if it's updated.

Better to have tagged releases for terraform-modules and invoke the tag in the HCL file:

module "vertica_prod_nodes" {
  source            = "[email protected]:myorg/terraform-modules.git?ref=v0.1.0//vertica/nodes"

This tip was caught and suggested by tflint, a Terraform linter which catches some additional mistakes that terraform plan doesn't.

Versioning the providers for the 0.10 release

This helps you avoid a warning from Terraform and pin specific versions of the provider modules.

In Terraform 0.10, providers were split from the main repo.

This means in provider.tf you need some versioning as follows:

provider "aws" {
  region = "${var.region}"

  version = "~> 0.1"
}

provider "null" {
  version = "~> 0.1"
}

provider "template" {
  version = "~> 0.1"
}

provider "terraform" {
  version = "~> 0.1"
}

Terraform will warn you about this.

*.auto.tfvars

If you have some shared var file, e.g.:

vertica-production/
├── nodes
│   ├── backend.conf
│   ├── main.tf
│   ├── provider.tf -> ../../provider_assumerole.tf
│   ├── shared.auto.tfvars -> ../shared.tfvars
│   ├── terraform.tfstate
│   ├── terraform.tfstate.backup
│   ├── terraform.tfvars
│   └── variables.tf
├── shared.tfvars
└── volumes
    ├── backend.conf
    ├── main.tf
    ├── provider.tf -> ../../provider_assumerole.tf
    ├── shared.auto.tfvars -> ../shared.tfvars
    ├── terraform.tfstate
    ├── terraform.tfvars
    └── variables.tf

Instead of symlinking ln -snf ../shared.tfvars ./shared.tfvars and invoking terraform with -var-file=./shared.tfvars - or, alternately, just invoking terraform -var-file=../shared.tfvars, you can symlink to shared.auto.tfvars and you can then omit the manual -var-file flag.

Terraform will automatically pick up *.auto.tfvars files in the cwd.

Extending tags in Terraform

Sometimes you want some default tags in a Terraform module but if the user wants additional tags, they should be able to define it.

Here’s how to do it with the merge and map interpolations:

variable "additional_ec2_tags" {
  type = "map"
}

aws_instance ... {
  [...]

  tags = "${merge(map(
      "Name",           "${var.name}",
      "Default1",       "${var.default1}",
      "Default2",       "${var.default2}",
    var.additional_ec2_tags)}"

  [...]
}

Rename a security group with Terraform

This is a difficult operation that often leads to a timeout due to an invalid lifecycle dependency.

The way to proceed is:

$ terraform state rm module.my_module.aws_security_group.old_security_group

This makes Terraform forget the old security group (but it’s not deleted).

After this, proceed to run a new terraform apply to create the renamed security group. Finally, you can manually clean up the old security group in the AWS console.

Comments