Thursday, July 28, 2016

Exposing a private Amazon RDS instance with iptables NAT rules

I needed to expose a private Amazon MySQL RDS instance to a 3rd party SaaS tool. I tried several approaches and finally found one that seemed to work pretty well.

I ended up creating a small EC2 instance in the same VPC as the RDS instance, and applied these iptables NAT/masquerading rules to it, mapping local port 3307 to port 3306 on the RDS instance, whose internal IP address is in this case 172.16.11.2.

# cat iptables_tunnel_port_3307.sh
#!/bin/bash

iptables -F
iptables -F -t nat
iptables -X
iptables -t nat -A PREROUTING -p tcp --dport 3307 -j DNAT --to 172.16.11.2:3306
iptables -A FORWARD -p tcp -d 172.16.11.2 --dport 3306 -j ACCEPT
iptables -t nat -A OUTPUT -p tcp -o lo --dport 3307 -j DNAT --to 172.16.11.2:3306
iptables -t nat -A POSTROUTING  -j MASQUERADE

I also had to enable IP forwarding on the EC2 instance:

# sysctl net.ipv4.ip_forward
# sysctl -p

At this point, I was able to hit the external IP of the EC2 instance on port 3307, and get to the private RDS instance on port 3306. I was also able to attach the EC2 instance to an EC2 Security Group allowing the 3rd party SaaS tool IP addresses to access port 3307 on the EC2 instance.

My thanks to the people discussing a similar issue on this thread of LinuxQuestions. Without their discussion, I don't think I'd have been able to figure out a solution.

Wednesday, July 13, 2016

Using JMESPath queries with the AWS CLI

The AWS CLI, based on the boto3 Python library, is the recommended way of automating interactions with AWS. In this post I'll show some examples of more advanced AWS CLI usage using the query mechanism based on the JMESPath JSON query language.

Installing the AWS CLI tools is straightforward. On Ubuntu via apt-get:

# apt-get install awscli

Or via pip:

# apt-get install python-pip
# pip install awscli

The next step is to configure awscli by specifying the AWS Access Key ID and AWS Secret Access Key, as well as the default region and output format:

# aws configure
AWS Access Key ID: your-aws-access-key-id
AWS Secret Access Key: your-aws-secret-access-key
Default region name [us-west-2]: us-west-2

Default output format [None]: json

The configure command creates a ~/.aws directory containing two files: config and credentials.

You can specify more than one pair of AWS keys by creating profiles in these files. For example, in ~/.aws/credentials you can have:

[profile profile1]
AWS_ACCESS_KEY_ID=key1
AWS_SECRET_ACCESS_KEY=secretkey1

[profile profile2]
AWS_ACCESS_KEY_ID=key2
AWS_SECRET_ACCESS_KEY=secretkey2

In ~/.aws/config you can have:
[profile profile1]
region = us-west-2

[profile profile2]
region = us-east-1

You can specify a given profile when you run awscli:

# awscli --profile profile1

Let's assume you want to write a script using awscli that deletes EBS snapshots older than N days. Let's go through this one step at a time.

Here's how you can list all snapshots owned by you:

# aws ec2 describe-snapshots --profile profile1 --owner-id YOUR_AWS_ACCT_NUMBER --query "Snapshots[]"

Note the use of the --query option. It takes a parameter representing a JMESPath JSON query string. It's not trivial to figure out how to build these query strings, and I advise you to spend some time reading over the JMESPath tutorial and JMESPath examples.

In the example above, the query string is simply "Snapshots[]", which represents all the snapshots that are present in the AWS account associated with the profile profile1. The default output in our case is JSON, but you can specify --output text at the aws command line if you want to see each snapshot on its own line of text.

Let's assume that when you create the snapshots, you specify a description with contains PROD or STAGE for EBS volumes attached to production and stage EC2 instances respectively. If you want to only display snapshots containing the string PROD, you would do:

# aws ec2 describe-snapshots --profile profile1 --owner-id YOUR_AWS_ACCT_NUMBER--query "Snapshots[?contains(Description, \`PROD\`) == \`true\`]" --output text

The Snapshots[] array now contains a condition represented by the question mark ?. The condition uses the contains() function included in the JMESPath specification, and is applied against the Description field of each object in the Snapshots[] array, verifying that it contains the string PROD.  Note the use of backquotes surrounding the strings PROD and true in the condition. I spent some quality time troubleshooting my queries when I used single or double quotes with no avail. The backquotes also need to be escaped so that the shell doesn't interpret them as commands to be executed.

To restrict the PROD snapshots even further, to the ones older than say 7 days ago, you can do something like this:

DAYS=7
TARGET_DATE=`date --date="$DAYS day ago" +%Y-%m-%d`

# aws ec2 describe-snapshots --profile profile1 --owner-id YOUR_AWS_ACCT_NUMBER--query "Snapshots[?contains(Description, \`PROD\`) == \`true\`]|[?StartTime < \`$TARGET_DATE\`]" --output text

Here I used the StartTime field of the objects in the Snapshots[] array and compared it against the target date. In this case, string comparison is good enough for the query to work.

In all the examples above, the aws command returned a subset of the Snapshots[] array and displayed all fields for each object in the array. If you wanted to display specific fields, let's say the ID, the start time and the description of each snapshot, you would run:

# aws ec2 describe-snapshots --profile profile1 --owner-id YOUR_AWS_ACCT_NUMBER--query "Snapshots[?contains(Description, \`PROD\`) == \`true\`]|[?StartTime < \`$TARGET_DATE\`].[SnapshotId,StartTime,Description]" --output text

To delete old snapshots, you can use the aws ec2 delete-snapshot command, which needs a snapshot ID as a parameter. You could use the command above to list only the SnapshotId for snapshots older than N days, then for each of these IDs, run something like this:

# aws ec2 delete-snapshot --profile profile1 --snapshot-id $id

All this is well and good when you run these commands interactively at the shell. However, I had no luck running them out of cron. The backquotes resulted in boto3 syntax errors. I had to do it the hard way, by listing all snapshots first, then going all in with sed and awk:

aws ec2 describe-snapshots --profile profile1 --owner-id YOUR_AWS_ACCT_NUMBER --output=text --query "Snapshots[].[SnapshotId,StartTime,Description]"  > $TMP_SNAPS

DAYS=7
TARGET_DATE=`date --date="$DAYS day ago" +%Y-%m-%d`

cat $TMP_SNAPS | grep PROD | sed 's/T[0-9][0-9]:[0-9][0-9]:[0-9][0-9].000Z//' | awk -v target_date="$TARGET_DATE" '{if ($2 < target_date){print}}' > $TMP_PROD_SNAPS

echo PRODUCTION SNAPSHOTS OLDER THAN $DAYS DAYS
cat $TMP_PROD_SNAPS

for sid in `awk '{print $1}' $TMP_PROD_SNAPS` ; do
echo Deleting PROD snapshot $sid
aws ec2 delete-snapshot --profile $PROFILE --region $REGION --snapshot-id $sid
done

Ugly, but it works out of cron. Hope it helps somebody out there.

July 14th 2016: I initially forgot to include this very good blog post from Joseph Lawson on advanced JMESPath usage with the AWS CLI.

Friday, July 01, 2016

More tips and tricks for running Gatling in Docker containers

This post is a continuation of my previous one on "Running Gatling tests in Docker containers via Jenkins". As I continued to set up Jenkins jobs to run Gatling tests, I found the need to separate those tests for different environments - development, staging and production. The initial example I showed contained a single setup, which is not suitable for multiple environments.

Here is my updated Gatling directory structure

gatling
gatling/conf
gatling/conf/production
gatling/conf/production/gatling.conf
gatling/conf/staging
gatling/conf/staging/gatling.conf
gatling/Dockerfile
gatling/results
gatling/user-files
gatling/user-files/data
gatling/user-files/data/production-urls.csv
gatling/user-files/data/staging-urls.csv
gatling/user-files/simulations
gatling/user-files/simulations/development
gatling/user-files/simulations/production
gatling/user-files/simulations/production/Simulation.scala
gatling/user-files/simulations/staging
gatling/user-files/simulations/staging/Simulation.scala

Note that I created a separate directory under simulations for each environment (development, staging, production), each with its own simulation files.

I also created a data directory under user-files, because that is the default location for CSV files used by Gatling feeders.

Most importantly, I created a separate configuration directory (staging, production) under gatling/conf, each directory containing its own customized gatling.conf file. I started by copying the gatling-defaults.conf file from GitHub to gatling/conf/staging/gatling.conf and gatling/conf/production/gatling.conf respectively.

Here is what I customized in staging/gatling.conf:

mute = true # When set to true, don't ask for simulation name nor run description
simulations = user-files/simulations/staging

I customized production/gatling.conf in a similar way:

mute = true # When set to true, don't ask for simulation name nor run description
simulations = user-files/simulations/production

Setting mute to true is important because without it, running Gatling in a Docker container was segfaulting while waiting for user input for the simulation ID:

Select simulation id (default is 'gatlingsimulation'). Accepted characters are a-z, A-Z, 0-9, - and _ 
Exception in thread "main" java.lang.NullPointerException
at io.gatling.app.Selection$Selector.loop$1(Selection.scala:127) at io.gatling.app.Selection$Selector.askSimulationId(Selection.scala:135) at io.gatling.app.Selection$Selector.selection(Selection.scala:50) at io.gatling.app.Selection$.apply(Selection.scala:33) at io.gatling.app.Gatling.runIfNecessary(Gatling.scala:75) at io.gatling.app.Gatling.start(Gatling.scala:65) at io.gatling.app.Gatling$.start(Gatling.scala:57) at io.gatling.app.Gatling$.fromArgs(Gatling.scala:49) at io.gatling.app.Gatling$.main(Gatling.scala:43) at io.gatling.app.Gatling.main(Gatling.scala)

The other customization was to point the simulations attribute to the specific staging or production sub-directories.

Since the CSV files containing URLs to be load tested are also environment-specific, I modified the Simulation.scala files to take this into account. I also added 2 JAVA_OPTS variables that can be passed at runtime for HTTP basic authentication. Here is the new Crawl object (compare with the one from my previous post):

object Crawl {
  val feeder = csv("staging-urls.csv").random

  val userName = System.getProperty("username")
  val userPass = System.getProperty("password")

  val crawl = exec(feed(feeder)
    .exec(http("${loc}")
    .get("${loc}").basicAuth(userName, userPass)
    ))
}

One more thing is needed: to make Gatling use a specific configuration file instead of its default one, which is conf/gatling.conf. To do that, I set GATLING_CONF as an ENV variable in the Dockerfile, so it can be passed as a 'docker run' command line parameter. Here is the Dockerfile:

# Gatling is a highly capable load testing tool.
#
# Documentation: http://gatling.io/docs/2.2.2/
# Cheat sheet: http://gatling.io/#/cheat-sheet/2.2.2

FROM java:8-jdk-alpine

MAINTAINER Denis Vazhenin

# working directory for gatling
WORKDIR /opt

# gating version
ENV GATLING_VERSION 2.2.2

# create directory for gatling install
RUN mkdir -p gatling

# install gatling
RUN apk add --update wget && \
  mkdir -p /tmp/downloads && \
  wget -q -O /tmp/downloads/gatling-$GATLING_VERSION.zip \
  https://repo1.maven.org/maven2/io/gatling/highcharts/gatling-charts-highcharts-bundle/$GATLING_VERSION/gatling-charts-highcharts-bundle-$GATLING_VERSION-bundle.zip && \
  mkdir -p /tmp/archive && cd /tmp/archive && \
  unzip /tmp/downloads/gatling-$GATLING_VERSION.zip && \
  mv /tmp/archive/gatling-charts-highcharts-bundle-$GATLING_VERSION/* /opt/gatling/

# change context to gatling directory
WORKDIR  /opt/gatling

# set directories below to be mountable from host
VOLUME ["/opt/gatling/conf", "/opt/gatling/results", "/opt/gatling/user-files"]

# set environment variables
ENV PATH /opt/gatling/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ENV GATLING_HOME /opt/gatling
ENV GATLING_CONF /opt/gatling/conf
ENV JAVA_OPTS ""

ENTRYPOINT ["gatling.sh"]

Finally, here is how I invoke 'docker run' to tie everything together:

docker run --rm -v ${WORKSPACE}/gatling/conf:/opt/gatling/conf -v ${WORKSPACE}/gatling/user-files:/opt/gatling/user-files -v ${WORKSPACE}/gatling/results:/opt/gatling/results -e GATLING_CONF="/opt/gatling/conf/staging" -e JAVA_OPTS="-Dusers=$USERS -Dduration=$DURATION -Dusername=myusername -Dpassword=mypass" /PATH/TO/DOCKER/REGISTRY/gatling

Note the GATLING_CONF parameter passed with -e with the value of /opt/gatling/conf/staging. Also note the username and password JAVA_OPTS parameters.

Happy load testing!

Modifying EC2 security groups via AWS Lambda functions

One task that comes up again and again is adding, removing or updating source CIDR blocks in various security groups in an EC2 infrastructur...