gmig - GCP migrations

pronounced as gimmick.

Manage Google Cloud Platform (GCP) infrastructure using migrations that describe incremental changes such as additions or deletions of resources. This work is inspired by MyBatis migrations for SQL database setup.

Introduction blog post

Your gmig infrastructure is basically a folder with incremental change files, each with a timestamp prefix (for sort ordering) and readable name.

/010_create_some_account.yaml
/015_add_permissions_to_some_account.yaml
/my-gcp-production-project
    gmig.yaml

Each change is a single YAML file with one or more shell commands that change infrastructure for a project.

# add loadrunner service account

do:
- gcloud iam service-accounts create loadrunner --display-name "LoadRunner"

undo:
- gcloud iam service-accounts delete loadrunner

A change must have at least a do section and optionally an undo section. The do section typically has a list of gcloud commands that create resources but any available tool can be used. All lines will be executed at once using a single temporary shell script so you can use shell variables to simplify each section. The undo section typically has an ordered list of gcloud commands that deletes the same resources (in reverse order if relevant). Each command in each section can use the following environment variables: $PROJECT,$REGION,$ZONE,$GMIG_CONFIG_DIR, and any additional environment variables populated from the target configuration (see env section in the configuration below).

State

Information about the last applied migration to a project is stored as a Google Storage Bucket object. Therefore, usage of this tool requires you to have create a Bucket and set the permissions (Storage Writer) accordingly. To view the current state of your infrastructure related to each migration, you can add the view section to the YAML file, such as:

# add loadrunner service account

do:
- gcloud iam service-accounts create loadrunner --display-name "LoadRunner"

undo:
- gcloud iam service-accounts delete loadrunner

view:
- gcloud iam service-accounts describe loadrunner

and use the view subcommand.

Conditional migration

Commands (do,undo,view) can be made conditional by adding an if section. You can only use custom environment variables and configuration parameters (PROJECT,ZONE,REGION) in expressions. If the expression evaluates to true then the do (up), undo (down) and view (view) commands are executed.

if: PROJECT == "your-project-id"
do:
- gcloud condig list

or with combinations:

if: (PROJECT == "your-project-id") && (ZONE == "my-zone")
do:
- gcloud condig list

For available operators, see Language-Definition

Help

NAME:
gmig - Google Cloud Platform infrastructure migration tool

USAGE:
gmig [global options] command [command options] [arguments...]

COMMANDS:
    init     Create the initial configuration, if absent.
    new      Create a new migration file from a template using a generated timestamp and a given title.
    up       Runs the do section of all pending migrations in order, one after the other.
             If a migration file is specified then stop after applying that one.
    down     Runs the undo section of the last applied migration only.
    down-all Runs the undo section of all applied migrations.
    plan     Log commands of the do section of all pending migrations in order, one after the other.
    status   List all migrations with details compared to the current state.
    view     Runs the view section of all applied migrations to see the current state reported by your infrastructure.
    force    state | do | undo
    util     create-named-port | delete-named-port
    export   project-iam-policy | storage-iam-policy
    help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
-q                   quiet mode, accept any prompt
-v                   verbose logging
--help, -h           show help
--print-version, -V  print only the version

Getting started

Installation

You need to compile it using the Go SDK.

go install github.com/emicklei/gmig@latest

init \

Prepares your setup for working with migrations by creating a gmig.json file in a target folder.

gmig init my-gcp-production-project

Then your filesystem will have:

/my-gcp-production-project/
    gmig.yaml

You must change the file gmig.yaml to set the Project and Bucket name.

# gmig configuration file
#
# Google Cloud Platform migrations tool for infrastructure-as-code. See https://github.com/emicklei/gmig.

# [project] must be the Google Cloud Project ID where the infrastructure is created.
# Its value is available as $PROJECT in your migrations.
#
# Required by gmig.
project: my-project

# [region] must be a valid GCP region. See https://cloud.google.com/compute/docs/regions-zones/
# A region is a specific geographical location where you can run your resources.
# Its value is available as $REGION in your migrations.
#
# Not required by gmig but some gcloud and gsutil commands do require it.
# region: europe-west1

# [zone] must be a valid GCP zone. See https://cloud.google.com/compute/docs/regions-zones/
# Each region has one or more zones; most regions have three or more zones.
# Its value is available as $ZONE in your migrations.
#
# Not required by gmig but some gcloud and gsutil commands do require it.
# zone: europe-west1-b

# [bucket] must be a valid GCP Storage bucket.
# A Google Storage Bucket is used to store information (object) about the last applied migration.
# Bucket can contain multiple objects from multiple applications. Make sure the [state] is different for each app.
#
# Required by gmig.
bucket: my-bucket

# [state] is the name of the object that hold information about the last applied migration.
# Required by gmig.
state: myapp-gmig-last-migration

# [env] are additional environment values that are available to each section of a migration file.
# This can be used to create migrations that are independent of the target project.
# By convention, use capitalized words for keys.
# In the example, "myapp-cluster" is available as $K8S_CLUSTER in your migrations.
#
# Not required by gmig.
env:
  K8S_CLUSTER: myapp-cluster

If you decide to store state files of different projects in one Bucket then set the state object name to reflect this, eg. myproject-gmig-state. If you want to apply the same migrations to different regions/zones then choose a target folder name to reflect this, eg. my-gcp-production-project-us-east. Values for region and zone are required if you want to create Compute Engine resources. The env map can be used to parameterize commands in your migrations. In the example, all commands will have access to the value of $K8S_CLUSTER.

new \