Use of AWS parameter store for storing the credentials, has made securing the applications much easier. You can encrypt the sensitive content like passwords, tokens and keys, then store them in parameter store and inject them to your application when they are needed.

Here is an example of the parameters stored there:

aws ssm describe-parameters

{
    "KeyId": "alias/parameter_store_key",
    "Name": "/my-service/my-password",
    "LastModifiedDate": 000.000,
    "Version": 2,
    "LastModifiedUser": "---",
    "Type": "SecureString",
    "Description": "1"
}
{
    "KeyId": "alias/parameter_store_key",
    "Name": "/my-second-service/my-password",
    "LastModifiedDate": 000,
    "Version": 1,
    "LastModifiedUser": "---",
    "Type": "SecureString",
    "Description": "1"
}

And you can decrypt them easily like this:

aws ssm get-parameter  --with-decryption --name /my-service/my-password

{
    "Parameter": {
        "Name": "/my-service/my-password",
        "LastModifiedDate": 000.000,
        "Value": "{{VALUE_REDACTED}}",
        "Version": 2,
        "Type": "SecureString",
        "ARN": "arn:aws:ssm:us-east-1:{{ACCOUNT}}:parameter/my-service/my-password"
    }
}

Behind the scenes, Parameter store uses a KMS key to do the encryption and decryption of the content. In our example above the KMS key used for the encryption/decryption is alias/parameter_store_key. The service or user that wants to access those contents in parameter store, not only should have access to parameter store but also should have access to the KMS key that is used for encryption/decryption.

Following the least privilege principal, it’s much recommended to give the permission only to the single key that is used. So you can ensure that service / user can only access the sensitive content that it has permissions to and avoid accidents.

Here is an example of an IAM role that provides that:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ssm:DescribeParameters",
                "ssm:GetParametersByPath",
                "ssm:GetParameters",
                "ssm:GetParameter"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "kms:Decrypt",
            "Resource": [
                "arn:aws:kms:us-east-1:{{ACCOUNT}}:key/ef1d677e-92ac-4710-9124-52f2844f15ee"
            ]
        }
    ]
}

As you can see, Parameter store is using the KMS key’s alias (alias/parameter_store_key), but the permission is defined based on the arn.

AccessDeniedException

I encountered an interesting issue in the past couple of days which I thought worths sharing.

Both parameters in the parameter store were using the same KMS key alias to encrypt and decrypt, and the first one was decrypting fine, but the second one was raising:

aws ssm get-parameter  --with-decryption --name /my-second-service/my-password

An error occurred (AccessDeniedException) when calling the GetParameter operation:
The ciphertext refers to a customer master key that does not exist,
does not exist in this region, or you are not allowed to access.
(Service: AWSKMS; Status Code: 400; Error Code: AccessDeniedException;)

when I changed the permission in the IAM role to key/* and basically allowing access to all the available keys, the exception was resolved and I could get the decrypted value of the encrypted content, as if it was using another key behind the scenes.

But again, that was not desired as we want to go with the least privilege access route.

It was becoming a mystery why two params, using the exact same key, one could be decrypted and one couldn’t.

To solve the mystery we referred to our good friend CloudTrail. To understand what is actually going on, and why one key could be decrypted and one not. As always our friend, CloudTrail, didn’t let us down.

By investigating further we realized that during our initial setup, we had created a KMS key first, assigned it the same alias alias/parameter_store_key, then created the parameter store entries. According to CloudTrail, the same key alias was later on deleted and then assigned to another KMS key. So although the parameter store shows the same KMS key alias for both params, but behind the scenes they were referring to two different keys. Which unfortunately there is no way to get the key ARN used by parameter store, it only shows the alias.

There we go, our theory that the second parameter needed access to a another KMS key was correct.

A simple resolution and safer to this is to rewrite the values in parameter store. When rewritten, it will use the latest KMS key, and for all the users of that entry in parameter store we can give them access to a single key.

On a side note, it is recommend to use Segment’s Chamber which is a wrapper around Parameter Store and makes interactions with it easier. It automatically injects the values as ENV VARs when needed.

If you have found this article interesting, you can follow me on twitter as I share my learnings there.