S3 Content Distribution via CloudFront Signed Urls Provisioned with AWS CDK

By Adam McQuistan in Security  04/01/2022 Comment

Introduction

In this article I demonstrate a Cloud Architecture pattern for distributing content stored in S3 but fronted by AWS CloudFront and accessed through signed URLs viable for a specified amount of time. I utilize NodeJS and the AWS SDK within AWS Lambda for generating the signed urls providing timeboxed access to files behind the CloudFront distribution in an S3 origin repository. To promote consistency of resource provisioning I've opted to define all AWS Resources and Services incuding the AWS Lambda implementation using AWS Cloud Development Kit (CDK). In this way a reader can easily redeploy the demo and experiment for further learning.

Overview of Solution Architecture

Below you can find a high-level diagram which shows the architecture components and the request / response flows of this solution.
The key components to this cloud architecture solution are as follows:

  • S3 Bucket to serve as a storage repository for files to be accessed via CloudFront Distribution
  • CloudFront Distribution which serves as the distribution frontend for the files in the origin S3 Bucket
  • S3 Object Lister Lambda is an AWS Lambda Function for NodeJS runtime which fetches the filenames stored in the S3 bucket
  • Url Signer Lambda is another AWS Lambda Function for NodeJS runtime that is invoked with a payload containing the name of a file in S3 which it also fetches a private key stored in AWS SSM Parameter Store to generate a CloudFront specific Signed Url to acces the target file saved in S3
  • SSM Parameter store contains the private key as an encrypted string which is used by Url Signer Lambda function to sign the access urls with
  • API Gateway serves as a simple REST API frontend that proxies the requests and responses to and from the previously mentioned AWS Lambdas

Deploying the Solution

The full code for this demo is available on GitHub so simply clone to follow along. To follow along you'll need to have NodeJS and AWS CDK installed as described in AWS CDK Docs along with the AWS CLI. You will also need git to clone the repo from GitHub and openssl installed to generate the private / public key pair needed to sign urls within the AWS Lambda function and to give to the AWS CloudFront distribution to decrypt and validate the signed urls being used to request content.
 
Clone repo and change directories into it.
git clone https://github.com/amcquistan/aws-cdk-presigned-url-rest-api.git
cd aws-cdk-presigned-url-rest-api
 
To generate public/private pem key pair first issue the following command using openssl specifying a RSA based key 2048 bits in length.
openssl genrsa -out private_key.pem 2048
If using Java to sign your cloudfront url you'll need to convert your PEM formatted private key to a DER format.
Use the openssl command like so to accomplish this.
openssl pkcs8 -topk8 -nocrypt -in private_key.pem -inform PEM -out private_key.der -outform DER
Use the next openssl commend to extract the public key from the previously generated key pair.
openssl rsa -pubout -in private_key.pem -out public_key.pem
Save the contents of these files into environment variables.
export PRIVATE_KEY=$(cat private_key.pem)
export PUBLIC_KEY=$(cat public_key.pem)
Save the private key contents in a SSM Parameter with encryption as a secure string.
aws ssm put-parameter \
  --name '/tci/demos/cloudfront/signedurls/privatekey' \
  --value $PRIVATE_KEY \
  --type SecureString
Can verify with the following.
aws ssm get-parameter \
  --name '/tci/demos/cloudfront/signedurls/privatekey' \
  --with-decryption
If necessary bootstrap the CDK application to your AWS account and region.
cdk bootstrap YOUR_AWS_ACCOUNT_ID/YOUR_AWS_REGION
Now build and deploy the CDK application.
npm run build
npm run deploy
The output of the above command will give the bucket name that was created along with the Api Gateway endpoint. Save these as they will be needed later. Here is example output from when I last deployed the app but understand that your output will be different.
 ✅  AwsCdkPresignedUrlRestApiStack

Outputs:
AwsCdkPresignedUrlRestApiStack.BucketName = awscdkpresignedurlrestapist-randofilesbkt3d6ff9c2-p2h3td503nh1
AwsCdkPresignedUrlRestApiStack.RestApiUrl = https://l4my74gn8b.execute-api.us-east-1.amazonaws.com/v1/

Testing the Functionality


Now that the solution has been deployed to the cloud in the prior step its a good time to put it through the paces and verify its functionality.

To start I'll create a couple of dummy text files to upload to S3 for this demo.
 
echo "I'm the first test file" > testfile1.txt  
echo "I'm the second test file" > testfile2.txt
aws s3 cp testfile1.txt s3://awscdkpresignedurlrestapist-randofilesbkt3d6ff9c2-p2h3td503nh1  
aws s3 cp testfile2.txt s3://awscdkpresignedurlrestapist-randofilesbkt3d6ff9c2-p2h3td503nh1 
Then I'll prove out that my List Objects REST API works using an HTTP client such as HTTPie, cURL, or even a browser.
http https://l4my74gn8b.execute-api.us-east-1.amazonaws.com/v1/s3objects -b
Example output.
[
    {
        "ETag": "\"5472b570c2e878e3adc891451b8d21d9\"",
        "Key": "testfile1.txt",
        "LastModified": "2022-03-31T22:40:01.000Z",
        "Size": 24,
        "StorageClass": "STANDARD"
    },
    {
        "ETag": "\"92c9265c51ade84a8bbc76f902e6f6ac\"",
        "Key": "testfile2.txt",
        "LastModified": "2022-03-31T22:40:12.000Z",
        "Size": 25,
        "StorageClass": "STANDARD"
    }
]
Then I'll select a file name to POST to the /urls endpoint effectively requesting a signed url to download the content with. For this I'll use testfile2.txt to generate a signed URL for.
http POST https://l4my74gn8b.execute-api.us-east-1.amazonaws.com/v1/urls file="testfile2.txt" -b
Example output.
{
    "signedUrl": "https://d3cvlqayyqxutd.cloudfront.net/testfile2.txt?Expires=1648807583&Key-Pair-Id=K1GJV5BZ7BHPXY&Signature=Idz8HwAIU41fUoVyrMms8nhHHlzXm8bzup6RViQuxCxBpYEqzAjEe~8aVxqrAr5v2HSyqx7DKxQSfiS6Adxq9R~PfQK-gcf3PoOfOuE-Gafgx2XkjUpqNspkS1fFBAx-i-PdAg8fn2lDHRAhneB2LKOaQJ3OtraPQssdSufa60x6lzAOxwwRTLemexBIFKb32ff8nGAjsoib8Y0~sbruTrCbZvShO~2VGkLOH7hyoBIPnvA~mhPExl-cBc7wUeYL0eNxqLuX-xLDlgnmPYV3glmqwcMTTvtlUhUk~V1vQofQjazisDWxDe2kq4hDKrIMVekRJHRgh3QIHYoiZos14g__"
}
Finally I can take the signed URL and fetch the file with my browser or some other HTTP Client.
http "https://d3cvlqayyqxutd.cloudfront.net/testfile2.txt?Expires=1648807583&Key-Pair-Id=K1GJV5BZ7BHPXY&Signature=Idz8HwAIU41fUoVyrMms8nhHHlzXm8bzup6RViQuxCxBpYEqzAjEe~8aVxqrAr5v2HSyqx7DKxQSfiS6Adxq9R~PfQK-gcf3PoOfOuE-Gafgx2XkjUpqNspkS1fFBAx-i-PdAg8fn2lDHRAhneB2LKOaQJ3OtraPQssdSufa60x6lzAOxwwRTLemexBIFKb32ff8nGAjsoib8Y0~sbruTrCbZvShO~2VGkLOH7hyoBIPnvA~mhPExl-cBc7wUeYL0eNxqLuX-xLDlgnmPYV3glmqwcMTTvtlUhUk~V1vQofQjazisDWxDe2kq4hDKrIMVekRJHRgh3QIHYoiZos14g__" -b
Example output.
I'm the second test file
To prove out that the underlying testfile2.txt resource is in fact protected I can issue the same request in un-signed form and expose the full headers like so.
http https://d3cvlqayyqxutd.cloudfront.net/testfile2.txt
Example output.
HTTP/1.1 403 Forbidden
Connection: keep-alive
Content-Length: 146
Content-Type: text/xml
Date: Thu, 31 Mar 2022 23:50:03 GMT
Server: CloudFront
Via: 1.1 c535ea37f0fd1edbebb6aafb708714a4.cloudfront.net (CloudFront)
X-Amz-Cf-Id: KUqqXOO8jm10svxejOto4ZgBu_jvBrKc1qp4J8IJiZSVkjFFQpyPjQ==
X-Amz-Cf-Pop: ORD56-P2
X-Cache: Error from cloudfront

<?xml version="1.0" encoding="UTF-8"?><Error><Code>MissingKey</Code><Message>Missing Key-Pair-Id query parameter or cookie value</Message></Error>
And you can see that the request failed.

Digging into the Details of the CloudFront and S3 Setup

Ok now that we have proved out that the solution provided works lets dig into it to understand how it works. The below diagram hones in on some of the specifics of the CloudFront and S3 resources.
I think the first thing to understand is that with the CloudFront Distribution is provisioned a relationship is established between it and the S3 bucket origin which permits the CloudFront identity to retrieve contents from the bucket. This is referred to Origin Access Identity (OAI). This is further enforced though an S3 Bucket Policy which locks down S3 GET Object access to just the specific CloudFront distribution as depicted below.
 
Ok so that locks down the access but now how do we go about granting access to those S3 files?

This is where the public / private keys come in to play. Back in the setup section I created a private key which I saved to AWS SSM Parameter Store and will use to generate a signed url specific to this CloudFront Distribution plus a S3 object key (aka filename). The public key gets added as a Key Group and associated with the CloudFront Distribution as shown in the diagram above. CloudFront will use this public key to decrypt and validate the signature of a URL targetting a specific S3 resource. If successfully validated CloudFront will then forward the request to S3 for the specified S3 object and return it back to the calling HTTP Client.
 
The below AWS CDK code snippet from lib/aws-cdk-presigned-url-rest-api-stack.ts shows how I generated the S3 Bucket along with the CloudFront Distrubtion and Key Group / Public Key Resources. It also shows how the relationships are established between the Keys, the CloudFront Distribtion along with the establishment of the Origin Access Identity between the S3 Bucket and CloudFront Distribution.
 
import { Stack, StackProps, CfnOutput } from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as s3 from "aws-cdk-lib/aws-s3";
import * as cloudfront from "aws-cdk-lib/aws-cloudfront";
import * as origins from "aws-cdk-lib/aws-cloudfront-origins";
import * as apigw from "aws-cdk-lib/aws-apigateway";
import * as lambda from "aws-cdk-lib/aws-lambda";
import * as iam from "aws-cdk-lib/aws-iam";

import * as path from 'path';


export interface PresignedUrlRestApiStackProps extends StackProps {
  readonly pubicKey: string;
  readonly privateKeySsmPath: string;
}

export class AwsCdkPresignedUrlRestApiStack extends Stack {
  constructor(
      scope: Construct,id:
      string,
      props: PresignedUrlRestApiStackProps
  ) {
    super(scope, id, props);

    const s3Bkt = new s3.Bucket(this, 'RandoFilesBkt');
    const pubKey = new cloudfront.PublicKey(this, 'SignedPubKey', {
      encodedKey: props.pubicKey
    });
    const cfDistribution = new cloudfront.Distribution(this, 'RandoFilesDistribution', {
      defaultBehavior: {
        origin: new origins.S3Origin(s3Bkt),
        trustedKeyGroups: [
          new cloudfront.KeyGroup(this, 'SignedKeyGroup', {
            items: [pubKey]
          })
        ]
      }
    });

    const api = new apigw.RestApi(this, 'RandoFilesApi', {
      description: 'REST API for Random Files in S3',
      deployOptions: {
        stageName: 'v1'
      },
      defaultCorsPreflightOptions: {
        allowHeaders: [
          'X-Api-Key',
          'Content-Type',
          'Authorization',
          'X-Amz-Date'
        ],
        allowOrigins: ['*'],
        allowCredentials: true,
        allowMethods: ['OPTIONS', 'GET', 'POST']
      }
    });

    const listObjsFn = new lambda.Function(this, 'ListRandoFilesFn', {
      runtime: lambda.Runtime.NODEJS_14_X,
      handler: 'index.handler',
      code: lambda.Code.fromAsset(path.join(__dirname, '..', 'lambdas', 's3-objects')),
      environment: {
        BUCKET_NAME: s3Bkt.bucketName
      }
    });
    const urlSignerFn = new lambda.Function(this, 'SignUrlFn', {
      runtime: lambda.Runtime.NODEJS_14_X,
      handler: 'index.handler',
      code: lambda.Code.fromAsset(path.join(__dirname, '..', 'lambdas', 'url-signer')),
      environment: {
        DISTRIBUTION_DOMAIN_NAME: cfDistribution.distributionDomainName,
        KEY_ID: pubKey.publicKeyId,
        PRIVATE_KEY_SSM_PATH: props.privateKeySsmPath
      }
    });
    urlSignerFn.addToRolePolicy(new iam.PolicyStatement({
      actions: ['ssm:GetParameter'],
      resources: ['*']
    }));
    s3Bkt.grantRead(listObjsFn);

    const s3ObjectApi = api.root.addResource('s3objects');
    s3ObjectApi.addMethod('GET', new apigw.LambdaIntegration(listObjsFn));

    const urlsApi = api.root.addResource('urls');
    urlsApi.addMethod('POST', new apigw.LambdaIntegration(urlSignerFn));

    new CfnOutput(this, 'RestApiUrl', {
      value: api.url
    });
    new CfnOutput(this, 'BucketName', {
      value: s3Bkt.bucketName
    });
  }
}
The above CDK Stack is then instantiated and passed the path to the private key SSM Param along with the contents of the public key sourced from an environment variable. This is accomplished as shown below from the bin/aws-cdk-presigned-url-rest-api-stack.ts file.
#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';
import {
  AwsCdkPresignedUrlRestApiStack
} from '../lib/aws-cdk-presigned-url-rest-api-stack';

const app = new cdk.App();
new AwsCdkPresignedUrlRestApiStack(app, 'AwsCdkPresignedUrlRestApiStack', {
  privateKeySsmPath: '/tci/demos/cloudfront/signedurls/privatekey',
  pubicKey: process.env.PUBLIC_KEY!
});

Digging into the Details of Generating Signed URLs for CloudFront Access to S3

Lastly I want to highlight the process of Signing a URL to allow access to one of the S3 objects for up to 12 hours.
The below code snippet is from the Url Signer Lambda function in lambdas/url-signer/index.js and accomplished the signing of urls for limited access to the S3 resources. This code integrates with AWS SSM Parameter store to source the private key then utilizes the NodeJS AWS SDK to create a signed url for the file name specified in the HTTP POST request proxyed through API Gateway.
const AWS = require('aws-sdk');

const ssm = new AWS.SSM();

async function fetchPrvateKey() {
  return new Promise(resolve => {
    ssm.getParameter({
      Name: process.env.PRIVATE_KEY_SSM_PATH,
      WithDecryption: true
    },
    (err, res) => {
      if (err) throw err;

      resolve(res.Parameter.Value);
    })
  });
}

exports.handler = async function(event, context) {
  console.log(JSON.stringify(event));

  const reqPayload = JSON.parse(event.body);

  const privateKey = await fetchPrvateKey();

  const signer = new AWS.CloudFront.Signer(process.env.KEY_ID, privateKey);

  const delta12Hrs = 12 * 60 * 60 * 1000;
  const signedUrl = signer.getSignedUrl({
    url: `https://${process.env.DISTRIBUTION_DOMAIN_NAME}/${reqPayload.file}`,
    expires: Math.floor((Date.now() + delta12Hrs) / 1000)
  });

  return {
    body: JSON.stringify({signedUrl}),
    statusCode: 201
  };
}

Conclusion

In this article I reviewed a solution for serving protected S3 content locking down distribution to timeboxed signed urls through the use of an AWS CloudFront Distribution and public/private key pair.
 
As always, I thank you for reading and please feel free to ask questions or critique in the comments section below.

Share with friends and colleagues

[[ likes ]] likes

Community favorites for Security

theCodingInterface