Deploying your Google AppEngine app with CircleCI
The Google Cloud CLI makes it simple to deploy a Google AppEngine app with one command (
gcloud app deploy), but there may come a time when you want to set up continuous integration with your app to take advantage of things like automatically running your tests or auto-deploying on merge.
For me, another big advantage of using CI - in my case, CircleCI - was being able to store secrets securely as environment variables in Circle. AppEngine allows you to specify environment variables in the deployment manifest (usually
app.yaml), but this would have meant hard-coding them and committing them in the repo, which is a big security no-no. We want things like API keys and client IDs/secrets to remain secret, because if someone got hold of these they could impersonate you or your app.
In this article I’m going to take you through:
- how to set up CircleCI for your repository
- environment variables in CircleCI (including Google service account credentials)
- dynamically injecting environment variables into
- deploying to Google AppEngine from CircleCI
- managing different deployment environments using CircleCI Contexts
Getting started with CircleCI
If you’re new to CircleCI, you can sign in with GitHub. By giving it access to GitHub, it’ll automatically look for your repositories and let you choose which ones to set up. Click the “Add projects” button, choose the repository for your AppEngine app, and click “Set Up Project”.
Select the language you’ve used - in my case, it was Node - and it’ll generate a sample CircleCI config file for you. This
config.yml file will live in the root of your project, in a folder called
.circleci (make sure you get the dot at the beginning). This is what CircleCI will look for in your repo to know what to do with your app. Create the folder, and follow the instructions that CircleCI gives you below the setup options to get started.
Create a Google Cloud service account
Before CircleCI can deploy your AppEngine app, we need to configure some environment variables, including your Google Service Account credentials that will allow Circle to deploy.
Create a new Service Account by going to the Google Cloud console and navigating to the Service Accounts panel. Create a new service account and download the key as JSON (this is important).
When it comes to granting permissions, I found the Project Editor role was the only one that worked - which made me a little uncomfortable from the point of view of the Principle of Least Privilege - but the deployments didn’t work when I gave it all the separate roles for Cloud Build and App Engine. (If anyone has a solution for this, I’d love to hear it.)
Setting environment variables in CircleCI
Now we’ll set the Google credentials as an environment variable, alongside the other env vars we’ll need for our app. In my case, it was a Slack app, so I needed to store the Slack OAuth client ID and secret.
From your project dashboard, click the cog to bring up the project settings, and then click “Environment Variables”.
Here, you can add in the name-value pairs you need. For my Slack app, that included a Slack Client ID, Client Secret, App ID and Signing Secret. Traditionally, env var names are written in
For the Google credentials, set an env var containing your project ID (you can find this from the “Project Info” panel in the Cloud Console) and one for your desired compute zone (e.g.
europe-west2). Next, we’ll create an environment variable containing your Google Service Account key.
Storing a Google Service Account key as an environment variable
Since the JSON key is a multi-line variable, we can’t just paste it into the value field. We’ll encode it using
base64 and store the result of that as the env var value as recommended in the CircleCI docs.
Open up a terminal, and enter the following command:
What this does:
cat reads the contents of the json file and then pipes it using the
| operator (feeds the result into) to the
base64 command. This will encode the contents of the key file, and then in turn pipes that to the system clipboard using
pbcopy. The encoded value will then be available to paste straight into CircleCI.
GCLOUD_KEY env var (or whatever you want to call it). Now we’re ready to configure the deployment.
Injecting environment variables into app.yaml
We’ll need to make a change to our
app.yaml file so that we can inject the env vars from CircleCI. In our Circle config we’re going to take advantage of the fact that the environment variables are available to us in bash, so we can reference them from a script. We’re actually going to turn our
app.yaml into a bash script.
Here’s my example
Rename the file to
app.yaml.sh and add the shebang at the top of the file. We’re then going to tell our script to
echo the contents of our app config - when
echo executes, it’ll evaluate the
$VARIABLE_NAMES, and pick up the values of those environment variables.
The quotation marks around the values are important for string values, even though they’re technically optional in yaml. My Slack Client ID was a string with the format
12345.67890 and because I’d left off the quotation marks, my app interpreted it as a number and rounded it up, resulting in an invalid Client ID.
Please note, once you’ve done this you won’t be able to manually deploy your app from the local CLI with this
app.yaml.sh file, unless you have these environment variables set locally.
Setting up CircleCI config.yml
config.yml that CircleCI provides for a Node app looks something like this, with a few more comments I’ve removed for brevity:
We’re going to tell Circle to use the
google/cloud-sdk docker image instead, as Google will be handling the actual building of our app, so CircleCI doesn’t need to know what version of Node we’re using.
If you’re planning on getting CircleCI to run your tests, you’ll want to keep in the part about downloading and caching dependencies, but for the purpose of this tutorial we’re going to take it out as we’re focusing on deployment.
Let’s rename the
build step to
deploy, and switch out the docker image. We’ll keep the working directory and the
checkout step as-is.
Next, we’ll copy across the
app.yaml by adding a
run step, executing our new
app.yaml.sh script and redirecting the result with
> (saving to a file) to a new
Authenticating with Google Cloud
Then, we need to get CircleCI to authenticate with Google Cloud. This is where we’ll decode our base-64 encrypted environment variable and use the Cloud SDK to authenticate.
echoing out the value of our
$GCLOUD_KEY variable, piping it into
base64 with the decode flag, and then piping the newly decoded value into the
gcloud auth activate-service-account command, with the flag
-key-file=-. This mysterious hyphen
- tells the command to use standard input (
stdin) rather than a given filename. In our case, standard input will be what we’ve piped into the command from
We’re then running commands to set the gcloud config (with the
--quiet flag so the output doesn’t fill up the CircleCI logs).
Finally, we can add a last step to deploy the project, using the same CLI command that you would have used manually.
Defining a workflow
Now you need to define a workflow at the bottom of
config.yml that will execute this step when you push to master.
This is a single-step workflow which will only execute on the
Commit and push your new
.circleci/config.yml files, and head to your CircleCI project dashboard. If everything worked, you should get a nice green build after a couple of minutes! If the build fails and you’re not sure why, you can check the Cloud Build logs (head to the StackDriver Logs Viewer and choose Cloud Build from the resource dropdown).
Managing different environments with CircleCI Contexts
If you want to have multiple deployed versions of your app - for example, a staging environment and a production environment - you’ll need different environment variables depending on where you’re deploying to. CircleCI Contexts allow you to define sets of environment variables that you can use in different steps of the workflow.
Contexts are defined at the organisation level, so head to your Organisation Settings page and open the “Contexts” panel.
Click the “Create Context” button, and give it a name (I’ve used ‘Example App Production’ for this tutorial). On the page for the context itself, you can add any environment variables your app needs. Any environment variables you defined earlier in the tutorial, you’ll need here - including Google Cloud credentials. I created a separate Google Cloud project for my staging application, so I had a different set of Google Cloud credentials for staging and production.
I created a context for production deployments, and kept the “default” set of environment variables for my staging deployments.
To use your new context, pass in a
context value to your workflow step, as follows:
This tells CircleCI to use the environment variables from the given context rather than the project-level ones. If you don’t include the
context field, it’ll pick up the environment variables you defined earlier in the project settings, so for my staging deployments I don’t specify a context.
Manual approval steps
I don’t want to automatically deploy to production, so I’ve put a manual approval step between my staging and production workflow steps. I can then prevent the
deploy-prod step from running unless the manual approval step has completed.
This workflow looks something like this:
Once you approve the manual step, the next step will kick off automatically.
You’re all set!
Now, you should be able to deploy your Google AppEngine app automatically via CircleCI and manage multiple deployment environments, storing your application secrets within CircleCI.
Any questions? Let me know on Twitter: @type__error