Perpetual AWS Authorization Using aws-vault’s Simulated ECS Metadata Server In a Container

Your local development services can be up and authorized indefinitely for AWS. Role chaining? MFA tokens? No problem! Use aws-vault’s ECS Metadata Server and take restarts out of your workflow.

Published in

FormSwift

15 min readApr 6, 2022

We use AWS for a variety of things here at FormSwift, including a few that are required for local development. We use aws-vault to safely manage individual identity accounts and only grant useful permissions to roles the individual developer assumes.

Additionally, for the past few months I’ve been working on revising our dev stack from a legacy hodgepodge of Vagrant boxes and separate Docker projects to a pseudo-monorepo using git submodules with one docker-compose.yml to rule them all (look for another article as this effort matures). One of the challenges I’ve faced in this project is that, while the straightforward aws-vault usage stays within a typical developer’s pain tolerance when applied to single services, using it for a whole bunch of services running in a single docker environment magnifies the inconvenience into a pretty poor developer experience.

In searching for a solution, I found a few thread comments here and there claiming it was possible, but precious little in the way of specifics. After spending a fair amount of time poring over the aws-vault source and chasing a few dead ends, I found an approach that works. I feel I would be remiss in not writing up the guide I couldn’t find a couple of weeks ago.

What is aws-vault?

The AWS CLI and assorted client libraries typically use environment variables to get their credentials, the most notable being AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. There are a few security concerns implied by the naive application of this pattern. The values are pretty impossible to remember, and annoying to copy/paste for every operation that needs them, which encourages people to do unsafe things like adding AWS credentials to their shell config. Additionally, when you use your own credentials directly, you’re typically providing them to code that is probably fine, but you also probably haven’t audited it for misuse of the credentials.

The aws-vault tool provides a few features that help to manage your credentials in a secure and consistent way. It achieves this with a few moving pieces:

Cooperation With AWS CLI: When you run aws-vault it uses the configuration in your ~/.aws directory to provide the information it needs about your account, roles and the like. You don’t even need the AWS CLI installed for aws-vault to consume its configuration files.

The Vault: Your real, persistent AWS credentials are added to a secure store, encrypted and password-protected. These are rarely exposed directly to a process. The vault can be backed by a number of different specific implementations. On a Mac, for instance, the default will be a Keychain. Because your credentials are in the vault, you do not need to add them to your account section in ~/.aws/credentials (and, in fact, should not). When using the AWS CLI by itself, your credentials file would often look like this:

[myaccount]
aws_access_key_id=AKIA****************
aws_secret_access_key=****************************************

When using aws-vault your credentials file can be as simple as this:

[myaccount]

Then instead of just running, say,aws s3 ls..., you will run aws-vault exec myaccount -- aws s3 ls.... Because aws-vault knows your credentials live elsewhere, it will prompt you for your vault password (if necessary) and find them there, then pass credentials on to the AWS CLI process as environment variables.

You can see what information aws-vault sends along to the consuming process like this:

$ aws-vault exec myaccount -- env | grep AWS

Temporary Credentials: By default, aws-vault automates the process of deriving temporary credentials from your real credentials using the GetSessionToken API of the AWS Security Token Service. These credentials are valid for twelve hours by default, but can be requested with validity ranging from fifteen minutes to 36 hours. It is these temporary session credentials that aws-vault uses for further operations. This reduces the severity of the breach in the unlikely event the consuming process does something bad that leaks the credentials it receives. You typically never see your actual account credentials again.

If for some reason you need your real credentials (read on to see one), aws-vault provides a way to bypass this automatic generation of temporary credentials using the --no-session flag. Suppose you ran this:

$ aws-vault exec --no-session myaccount -- env | grep AWS

You would see that the AWS credential environment variables are set to your actual key id and secret key. This, of course, disables some of the security aws-vault affords you, and should not be used without considering other ways of achieving what you want first and auditing what you do with the credentials once you have them.

Automatic Role Assumption: Your AWS environment may use roles your account can assume to acquire specific permissions, and aws-vault will read those roles out of your ~/.aws/config file and make them available. As a simplified example, your config file might look like this:

[default]
region = us-west-1[profile myaccount]
mfa_serial = arn:aws:iam::012345678901:mfa/me[profile local-developer]
source_profile = myaccount
role_arn = arn:aws:iam::012345678901:role/LocalDeveloperRole
mfa_serial = arn:aws:iam::012345678901:mfa/me

When you run aws-vault exec local-developer -- ... the local-developer profile will be dereferenced back through the temporary credentials and ultimately to your real credentials in the vault, so that the credentials the consuming process receives represent you operating under your organization’s LocalDeveloperRole.

This will also handle prompting for your MFA token as needed. By default aws-vault will prompt at the terminal, but there are other supported ways of providing the token. The --mfa-token (or -t) flag to aws-vault will let you supply the token on the command line. You can also add a credential_process that lets aws-vault prompt you with (among other options) a GUI popup. For example, on a Mac, you can add this line to your credentials or config file sections:

credential_process = aws-vault exec identity --json --prompt=osascript

This will invoke aws-vault's prompting mechanism to get the MFA token, which in turn we have configured to use osascript to pop up a Mac input dialog. This is handy for some IDE plugins that can’t prompt at a terminal.

Here there be dragons

For all the benefits aws-vault provides, it combines with some AWS behaviors to pose some significant problems for keeping a service running with authorization for long periods:

The temporary credential returned by the GetSessionToken API can have a relatively long lifetime, but is always limited.
Likewise, the AssumeRole API also returns a time-limited credential.
Eventually, in a few hours, AWS will want you to provide your MFA token again to prove you can get new credentials.
Even more restrictively, when you use a temporary credential to assume a role, you get only get one hour to work before your credentials go off. No, your friendly neighborhood devops engineer did not configure this short timeout because they hate you and want you to be sad. AWS considers this situation role-chaining and imposes the hard one-hour limit itself. It cannot be extended.

You can partly get around the one-hour limit using --no-session, but the other limits still apply, so if you aren’t restarting every hour, you’re at best restarting every other day or so. For someone (like me) who likes to keep a full environment running in the basement to offload resource use from the laptop, even this is a bit of a pebble in the shoe.

To infinity and beyond

Getting around the time limits here requires a few lesser-known features of aws-vault. There is a partial solution in the contrib/_aws-vault-proxy directory of the aws-vault repo. It’s similar to what I came up with in many ways (like, it probably would have saved me some time if I’d looked in the contrib directory earlier), but uses a little Go utility to do the work.

aws-vault as a service

There is an option to run aws-vault in a mode that emulates the metadata servers for either EC2 or ECS. In real AWS, the metadata servers are how your application gets information about its runtime environment, including its authorization.

You can get an EC2-style server with --server or an ECS-style server using --ecs-server. Because I’m working in a Dockerized environment, and because it seemed to offer a little more flexibility, the ECS server seemed the right choice here.

Once running, the server will dutifully spit out credentials on demand. Instead of providing AWS credentials directly to the application, you give the server URL using the AWS_CONTAINER_CREDENTIALS_FULL_URI environment variable. Many (but not all) AWS client libraries (including boto3 used in our Python services) already understand and respect this environment variable.

Here, too, there are complications. The ECS server binds to a random free port, and requires a random token in an Authorization header to work, which is provided in the AWS_CONTAINER_AUTHORIZATION_TOKEN variable. If aws-vault is running in a docker-compose service, the environment variables it supplies to its child process are hard to communicate to sibling services.

Authorization by proxy

To work around the environment variable issues, a reverse proxy was the silver bullet. In my implementation I chose tinyproxy due to its small size and ease of configuration for small-scale applications. The configuration for tinyproxy does not support environment variables, but even given that limitation it was easier to use a script that writes out the configuration and then launches the proxy than to use another solution.

Here, then, is my proxy.sh:

#!/usr/bin/env bashcat > proxy.conf <<EOF
Port 80
Listen 0.0.0.0
AddHeader "Authorization" "${AWS_CONTAINER_AUTHORIZATION_TOKEN}"
ReversePath "/" "${AWS_CONTAINER_CREDENTIALS_FULL_URI}"
ReverseBaseURL "http://169.254.170.2/"
ReverseOnly Yes
EOFtinyproxy -c proxy.conf -d

The service is thus exposed on the container’s HTTP port, which is proxied to aws-vault's random-port URL, and we inject the correct Authorization header so that other services do not need to know the token.

Note the hardcoded link-local IP address in the ReverseBaseURL. Although you could give this a domain name within the docker-compose network, boto3 (and maybe others — I haven’t checked) will only connect to metadata servers on 169.254.170.2 or 127.0.0.1, and the IP must be provided as such. A DNS name will be rejected even if it resolves to a supported IP address.

The authorization token is optional, and the header will only be sent from the client if the environment variable is set. Therefore, to give your other services access, you only need something like this in your docker-compose service definition:

environment:
  AWS_CONTAINER_CREDENTIALS_FULL_URI: http://169.254.170.2/

The client will see the URL variable set and hit the aws-vault service on port 80 to request a set of credentials, and the request will be proxied, with the authorization token added, to the actual random-port server running in the container.

You don’t know me

Now we’ve got a service to provide credentials, but we still need to get it authorized in the first place.

For the service to be able to generate a stream of updated credentials, it needs access to your real AWS credentials. One way to do this is by using --no-session:

$ aws-vault exec --no-session myaccount -- docker-compose up -d

This will supply your persistent AWS credentials to the docker-compose environment, and you will forward the credential variables into the aws-vault service. I supply them as build args because we can do most of the setup at build time and thus avoid having the credentials obviously visible in the container’s environment when you exec in. In docker-compose.yml:

services:
  aws_vault_server:
    build:
      context: ./aws_vault_server
      args:
        - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID:-}
        - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY:-}
        - AWS_USER=${AWS_USER:-}
    container_name: aws_vault_server
    networks:
      local_addr:
        ipv4_address: 169.254.170.2  my_authorized_service:
    ...
    environment:
      AWS_CONTAINER_CREDENTIALS_FULL_URI: http://169.254.170.2/       
    networks:
      my_app_network:
      local_addr:...networks:
  my_app_network:
  local_addr:
    ipam:
      driver: default
      config:
        - subnet: 169.254.170.0/24
...

Notice how we define a network to serve the link-local address range the service needs to live in, and then explicitly assign the correct IPv4 address to the service in that network. The service that needs credentials will then need to be given the credentials URI as an environment variable, and will need to be attached to the link-local network in addition to the overall application network.

Then in aws_vault_server/Dockerfile:

FROM archlinux
ARG AWS_ACCESS_KEY_ID
ARG AWS_SECRET_ACCESS_KEY
ARG AWS_USER
ENV AWS_VAULT_BACKEND="file"
ENV AWS_VAULT_FILE_DIR="/root/.aws-vault"
ENV AWS_VAULT_FILE_PASSPHRASE="XXXXXXXXXXXX"
COPY files/ /
RUN AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
    AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
    AWS_USER=${AWS_USER} \
    /provision.sh
ENTRYPOINT ["aws-vault", "exec", "--ecs-server", "--debug", "local-developer", "--", "/proxy.sh" ]

The provisioning script is called with the ARGs supplied as environment variables at build time. A few other points worth noting:

The AWS_USER argument is one I’ve defined to provide your AWS username, which is needed to set up MFA serials.
I’m using archlinux as my base image because pacman has all the dependencies I’ll need for this container, including aws-vault, with minimal fuss.
I’m configuring the container’s internal aws-vault to use the file vault backend, which is just a plain encrypted file supported on all platforms.
For now I’m using a hardcoded password for the vault, but plan to tighten security a bit by generating a random password at build time. Keep in mind, though, that this is intended only for local development, and that anybody with access to the insides of your local container can also read the environment of the process pretty easily anyway.
The ENTRYPOINT here shows how we launch the proxy inside the container, with an ECS server, under the local-developer role.
NOTE: As-is, this won’t quite launch. It will complain that --ecs-server is not compatible with --prompt=terminal(the default). Don’t worry, we’re not done with this yet.

Provisioning the aws-vault server container

The provision.sh script called by the Dockerfile does some vital things:

#!/usr/bin/env bash

# install dependencies
pacman -Sy --noconfirm --needed \
  aws-vault \
  tinyproxy \
  which

# create config dirs
mkdir /root/.aws
mkdir /root/.aws-vault

# import aws config and add identity to vault from env
cat /tmp/credentials | sed "s/mfa\/$/mfa\/${AWS_USER}/" > /root/.aws/credentials
cat /tmp/config | sed "s/mfa\/$/mfa\/${AWS_USER}/" > /root/.aws/config
aws-vault add --env myaccount

First the script installs dependencies we need. The first two are obvious, but which is not included in Arch’s docker image, and is required by some other things we’ll be running.
We create .aws and .aws-vault directories in root's home directory. The .aws folder is similar to the one in your own home directory. The .aws-vault directory is just a convenient place to stash the file-backed credential vault.
We could certainly use the local ~/.aws, but I opted to create a more constrained version of the config that only contains the local-developer role profile that the aws-vault service is meant to run. To make it generic to the user, I stripped the username off the mfa_serial values and then I just append ${AWS_USER} again at build time to create a full ARN.
We install AWS credentials into the container’s file vault using the --env flag to aws-vault add. This causes the access key id and secret access key to be read from the standard environment variables instead of prompting the user.

As you can see from the ENTRYPOINT above, there is no longer any need to use --no-session inside the container. In an hour, when the current role credentials expire, aws-vault will just re-request your real credentials from the vault, generate a new session token, and be good for another hour. We’ve got the vault password in the environment, so no prompts for that either.

At this point we have a container that will continue generating credentials right up until aws-vault decides it needs your MFA token again, but that’s going to be relatively soon too, so there’s still one more thing to fix.

Here’s my number, so MFA me

You’ll recall I wrote earlier that there is a variety of “prompt” implementations available for aws-vault. You might be inclined to look for a way to use an environment variable or a script to get the token, and indeed it’s possible there’s a way to do it with credential_process, but I got an alternate solution working first.

One of the prompt methods aws-vault supports is pass, which is an open and highly Unix-ish CLI password manager backed by a GPG-encrypted store. Furthermore, there is a TOTP extension for pass, and aws-vault supports it. All we need to do is add our MFA seed to the pass vault and aws-vault can call that every time it needs to provide a new token.

Strangely enough, pass seems to be supported only for the MFA case, and not as a vault backing store, so we’ve got two credential stores going on in this container. Odd, but it works. The packages we need for this are also available in the Arch Linux pacman repo, so we only need a few modifications to get this working. In provision.sh:

# install dependencies
pacman -Sy --noconfirm --needed \
  aws-vault \
  pass \
  pass-otp \
  tinyproxy \
  which...# create the pass vault and insert the MFA serial
gpg --quick-gen-key --yes --batch --passphrase '' aws-vault
pass init aws-vault
pass otp insert aws-vault <<< ${AWS_MFA_URI}

In the Dockerfile for the vault service:

FROM archlinux
ARG AWS_MFA_URI...ENV AWS_VAULT_PROMPT="pass"
ENV PASS_OATH_CREDENTIAL_NAME="aws-vault"...RUN AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
    AWS_MFA_URI=${AWS_MFA_URI} \
    AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
    AWS_USER=${AWS_USER} \
    /provision.sh...

And then in docker-compose.yml:

aws_vault_server:
  build:
    context: ./aws_vault_server
    args:
      - AWS_MFA_URI=${AWS_MFA_URI:-}...

You’ll need to supply this environment variable (in addition to the credentials provided by aws-vault when building the vault service).

The AWS_MFA_URI value needs to be an otpauth URI, which you might have directly for some services, but if you only have the bare serial value (which, for AWS, you probably will), you can reconstitute a URI like this:

export AWS_MFA_URI="otpauth://totp/aws-vault?digits=6&secret=${AWS_MFA_SERIAL}&algorithm=SHA1&issuer=AWS&period=30"

You’re probably already using an MFA app (like Authy or Google Authenticator) to provide your MFA tokens, so you’ll need to reset your MFA to get your hands on the secret value. To do this:

Log in to your organizational AWS account. If you’re like us, managing your MFA token is one of the few things you’ll have direct authorization to do.
In the console’s top-right dropdown, select “Security credentials.”
On the resulting “My security credentials” page, scroll down to the “Multi-factor authentication (MFA)” section and click “Manage MFA device.”
Remove and recreate your MFA device. Record the generator secret before adding it back to your MFA app of choice.
This value will be the value of AWS_MFA_SERIAL in the URI construction block above.

Tying it all together

That’s pretty much it. To get the vault server running, you’d need to do something like this:

$ AWS_MFA_SERIAL="<your mfa secret>" aws-vault exec --no-session myaccount -- docker-compose build aws_vault_server

When this container comes up and you attach other services to it, it will continue to provide up-to-date credentials with no prompting for as long as the service is running.

Summary

To sum up, I’ll provide the complete example files we built up over the course of this article.

The docker-compose command above invokes your project’s docker-compose.yml:

version: '3.9'
services:
  aws_vault_server:
    build:
      context: ./aws_vault_server
      args:
        - AWS_MFA_URI=${AWS_MFA_URI:-}
        - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID:-}
        - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY:-}
        - AWS_USER=${AWS_USER:-}
    container_name: aws_vault_server
    networks:
      local_addr:
        ipv4_address: 169.254.170.2  my_authorized_service:
    image: my_service_image:latest
    environment:
      AWS_CONTAINER_CREDENTIALS_FULL_URI: http://169.254.170.2/       
    networks:
      my_app_network:
      local_addr:networks:
  my_app_network:
  local_addr:
    ipam:
      driver: default
      config:
        - subnet: 169.254.170.0/24

The aws_vault_server service invokes the corresponding Dockerfile:

FROM archlinux
ARG AWS_MFA_URI
ARG AWS_ACCESS_KEY_ID
ARG AWS_SECRET_ACCESS_KEY
ARG AWS_USER
ENV AWS_VAULT_BACKEND="file"
ENV AWS_VAULT_FILE_DIR="/root/.aws-vault"
ENV AWS_VAULT_FILE_PASSPHRASE="XXXXXXXXXXXX"
ENV AWS_VAULT_PROMPT="pass"
ENV PASS_OATH_CREDENTIAL_NAME="aws-vault"
COPY files/ /
RUN AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
    AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
    AWS_USER=${AWS_USER} \
    AWS_MFA_URI=${AWS_MFA_URI} \
    /provision.sh
ENTRYPOINT ["aws-vault", "exec", "--ecs-server", "--debug", "local-developer", "--", "/proxy.sh" ]

During build, the Dockerfile runs provision.sh:

#!/usr/bin/env bash

# install dependencies
pacman -Sy --noconfirm --needed \
  aws-vault \
  pass \
  pass-otp \
  tinyproxy \
  which

# create config dirs
mkdir /root/.aws
mkdir /root/.aws-vault

# import aws config and add identity to vault from env
cat /tmp/credentials | sed "s/mfa\/$/mfa\/${AWS_USER}/" > /root/.aws/credentials
cat /tmp/config | sed "s/mfa\/$/mfa\/${AWS_USER}/" > /root/.aws/config
aws-vault add --env myaccount# create the pass vault and insert the MFA serial
gpg --quick-gen-key --yes --batch --passphrase '' aws-vault
pass init aws-vault
pass otp insert aws-vault <<< ${AWS_MFA_URI}

In addition to adding the auth secrets into local vaults, this will copy the basic config and credentials files into the appropriate location, while putting your AWS_USER into the mfa_serial values.

credentials:

[myaccount]
mfa_serial = arn:aws:iam::012345678901:mfa/
credential_process = aws-vault exec identity --json --prompt=pass

config:

[default]                                                                                                                                    
region = us-west-1                                                                                                                           
                                                                                                                                             
[profile myaccount]                                                                                                                           
mfa_serial = arn:aws:iam::012345678901:mfa/
                                                                                                                                             
[profile local-developer]                                                                                                                    
role_arn = arn:aws:iam::012345678901:role/LocalDeveloperRole                                                                             
source_profile = myaccount                                                                                                                      
mfa_serial = arn:aws:iam::012345678901:mfa/

At runtime, the aws_vault_server container will execute:

aws-vault exec --ecs-server --debug local-developer -- /proxy.sh

This launches an ECS-style metadata server using the vault-stored credentials for both keys and MFA, and then provides the server’s AWS_CONTAINER_CREDENTIALS_FULL_URI and AWS_CONTAINER_AUTHORIZATION_TOKEN in the environment for /proxy.sh:

#!/usr/bin/env bash

cat > proxy.conf <<EOF
Port 80
Listen 0.0.0.0
AddHeader "Authorization" "${AWS_CONTAINER_AUTHORIZATION_TOKEN}"
ReversePath "/" "${AWS_CONTAINER_CREDENTIALS_FULL_URI}"
ReverseBaseURL "http://169.254.170.2/"
ReverseOnly Yes
EOF

tinyproxy -c proxy.conf -d

Every time my_authorized_service needs credentials for an operation, it will send a request to http://169.254.170.2/. This request will be forwarded to the internal metadata server, which will (as needed) construct new role credentials based on your stored AWS credentials, and request a current MFA token from pass, and then return a JSON blob containing a current, valid set of service credentials within LocalDeveloperRole.

Other services that need local developer access will therefore always have it as long as aws_vault_server is running, QED.

I got the sense while browsing for this solution myself that this was not obvious to a whole lot of people. It certainly wasn’t to me. I hope this helps your dev teams get maximum bang for their aws-vault buck by reducing time spent hassling over credentials for local development.

And in case it isn’t obvious, don’t try to use this in a non-local context. It’s somewhat secure, but the protections are a screen door at best against a serious attacker. Your AWS deployments already have a real metadata server available to them.