Terraform: chicken/egg problem

So you’re starting to work with Terraform and would like to store the state in S3 bucket. Sounds good but how are you going to create the S3 bucket? You don’t want to use AWS Console to do that — you decided to have IaC and having first step breaking the rule is awkward at least. You can find a few articles and ideas how to deal with that and here I would like to present approach I tried.

What are our goals?

  • No manual operations through AWS Console
  • All resources managed by Terraform (includes state file in S3 bucket)

Step 1 — configure Terraform

Let’s start with our main.tf file to configure provider and backend. The problem here is that for terraform block you cannot use resources or even variables.

So let’s try another approach. We can define our Terraform backend as s3 and do not provide required variables — we will do that later. Additionally we have standard definition of provider and two modules:

  • backend — it will define our infrastructure for storing Terraform’s state.
  • infrastructure — it will define infrastructure required by our service.

Here is our main.tf file:

We have one variable to define configuration of our bucket defined in variables.tf like this:

Now let’s take a look on our backend module — modules/backend/main.tf

Nothing special — we’ve got our AWS resource and we would like to turn on versioning (as HashiCorp advise).

Step 2 — try & solve issues

Let’s try to run terraform init and provide required variables (bucket and key). As you can expect it fails:

Bucket doesn’t exist. Let’s fix it and create bucket using AWS CLI: aws s3api create-bucket --bucket ${BUCKET} --region ${AWS_REGION} --create-bucket-configuration LocationConstraint=${AWS_REGION} (don’t forget to export AWS credentials). So bucket exists, let’s try to init our configuration once — we can provide backend configuration in command line (we have new environment variable SERVICE_NAME which defines the filename in bucket to store state):

And…

Great! But Terraform doesn’t know about our backend S3 bucket. So for example it won’t enable versioning for our bucket (and we specified that in our modules/backend/main.tf file). How we can solve that? By importing resource: terraform import module.backend.aws_s3_bucket.tfstate_bucket ${BUCKET}. Now you should see nice message: Import successful!

Step 3 — verify

Let’s do our terraform plan :

We can see that Terraform wants to modify our resource to enable versioning. Everything looks as expected.

Summary

With that approach we have all resources managed by Terraform. We can set appropriate permissions etc. And what’s even more important we will avoid any manual configuration or complex scripts. This a few steps we can wrap into nice script (you can make it easily nicer and more verbose — please share in comments your version :) ):

The whole repository can be found here: https://github.com/mmatecki/tf-s3-state

It’s my first article on Medium.com comments are more than welcome!

Tech Lead / Senior Software Engineer @ Zoopla

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store