#terraform your #fastly Service - a simple example

Dec 2, 2018 fastly aws terraform english gitlab
Share on:

In the past 2,5 years while working for Ticketmaster in London, I had the chance to use extensively (still do) Fastly, one of the best CDN and Edge Compute services out there. Fastly is not just a web cache but a wholistic Edge service offering. It enables integrators (users) to actually bring their application (web service/site) closer to their end users and at the same time offload some of the logic and decision making that every web application has to do, from the main origins (your servers) and push it to the edge (a simplified explanation of edge computing).

Intro

The post is mostly a follow up (I know it took some time) of my short talk on Fastly EU Altitude day here in London. You can find the video here. One of the problems we were facing at Ticketmaster is the extremely big number of services (domains) owned by different teams and parts of the company. Some times a website might span 3-5 different individual Fastly Configs e.g. you have the dev site, the prod site, the staging site, or a slightly experimental version of the site, on all of them you maintain a different set of origins and you want to _proxy them through fastly in order to test end to end the integration etc.

To cut a long story short, the number of domains (services) was getting big very big. In the early adoption days, most of the teams and individuals would login to the web console of Fastly and would start updating configs (like you would do with AWS yeas ago). Some other teams would use things like Ansible and some custom integrations. I was always overwhelmed by both approaches. I could see that having 200 different configs and people manually editing them was not going to scale, at the same time be-spoke integrations with different tools felt very complex +not out of the box support. After some time I discovered the Fastly Terraform Provider and I think this was like a milestone, we could finally, treat our Fastly configs as code, push them to repos, introduce CI/CD practises and have a very good grip on who is doing what at any given time with our services.

The example

You can find the example here. My main motivation is not to show you how you do Terraform, but to provide a simple project structure (is dead easy) and how the terraform provider can be applied. Also don't worry about the specifics, this example is about having a personal domain (www.javapapo.eu) which I own, and then serving using an S3 bucket some static content. You don't have to do the same, the domain could be your company's team domain and the backends do not have to be some simple S3 buckets.

The main points of this example are:

  • Provide a sample structure to start with
  • Illustrate the basics of the Fastly terraform provider
  • Illustrate how you can add and upload custom VCL to your terraformed Fastly Service

Pre-Req

You will need a Fastly account. Once you activate your account you will need to create a new an API TOKEN see here. This token will be used by the Terraform provider, to talk to Fastly.

You will need to export or provide to terraform this key!

1export _FASTLY_API_KEY=XXXXXXXXXXXXXXXXXXXX'

Also you might notice that in my example I use a private S3 bucket to store my terraform state. You don't have to do the same, especially if you plan experimenting first. So you can remove the section where I configure a backend for the terraform provider.

In case you want to use an S3 bucket you will need to provide to 'terraform_ the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY on your environment.

1export AWS_DEFAULT_REGION=YOUR_REGION; 
2export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_KEY; 
3export AWS_ACCESS_KEY_ID=YOUR_SECRET_KEY_ID; ```

In case you want to 100% replay the example, you need your own domain, attach a Fastly Service to the domain, use an S3 bucket the. So I had to:

  • I registered a domain on AWS, _example : www.javapapo.eu'
  • I added a CNAME on my domain through the AWS console to point to Fastly (NON SSL) see here
  • I created an S3 bucket (with the appropriate policies, so I can serve some static html)

About the example

As you can see the structure of the project is quite flat. You have a couple of terraform files and then a subfolder with the custom VCL that we want to upload to our service. The Fastly terraform provider is actually a mirror, of the properties and settings you can configure manually through the Dashboard. If you are a newcomer I would just you open a browser tab on Fastly's Dashboard, then another one on the documentation of the provider see here so that you can see how each section in terraform correlates with the real thing. It is actually ok if you start experiment with Fastly using the console (dashboard). Once you gain some experience then you move to automating most of the configs that you had applied manually int he past.

Till now the most frequent question I get is related to combining Terraform and VCL together. On very advanced scenarios you will want to take advantage of the capability of fastly to be a platform where you can take smart decisions based on the incoming requests and actually offloading some processing time from your origins. The more logic you add the more you offload logic (but watch out - not to overcomplicate things, meaning your origins will still be the source of your business logic). Fastly can help you on executing some of this web related logic, like make redirects, make smart caching, load balance through the use of a somewhat primitive C like language called VCL. For those that have extensive experience with Varnish, well we are talking about Varnish 2.x VCL code.

Attaching to your terraform settings, snippets like the following:

1{ name = "Main" content = "${file("${path.module}/vcl/main.vcl")}" main = true }

You actually attach to your service pieces of your VCL code. Fastly will use all this code and will combine them with the standard code being executed for you per request. Here is an example on the provided main.vcl of my service where I do a redirect to my personal blog (here), when you try to hit the url https://www.javapapo.eu/blog

 1sub vcl_recv {
 2 # if you want my blog- redirect to it! 
 3 if(req.url.path ~ "^/blog"){
 4   error 666; 
 5 } 
 6 #FASTLY recv 
 7 }
 8
 9 sub vcl_error { 
10 if (obj.status == 666) { 
11   set obj.status = 307; 
12   set obj.http.Cache-Control = "no-cache"; 
13   set obj.response = "Moved Temporarily"; 
14   set obj.http.Location = "https://javapapo.blogspot.com";
15   return (deliver); 
16 } 
17 }

This is a typical example where I leveraged Fastly to make a smart decision upon an incoming request and perform a redirect without letting the request land on my origins or web servers. You can find the reference documentation of Fastly VCL here. So have a look on the example and use it a starting point Happy Fastly-ing

Repo link : https://gitlab.com/javapapo-public/fastly-tf-example