Using Pre-Signed URLs to Make Requests to AWS

Figure showing GET and POST requests sent to the S3 and Secrets Manager services using pre-signed URLs.

When you send HTTP requests to AWS, you sign the requests so that data is protected in transit, and AWS can verify the identity of the requester. If you use the AWS Command Line Interface (CLI) or AWS SDKs, the requests are signed for you automatically.

To sign a request, you calculate a hash known as the signature. Then, you add the signature to the HTTP Authorization header or the query string of the request. In the latter case, the signature is part of the URL and, thus, this type of URL is called a pre-signed URL.

The pre-signed URLs are often used with the Amazon S3 service. For example, you can provide your customer with a pre-signed URL that allows the customer to upload a specific object to your private S3 bucket. In this case, you do not need to require the customer to have AWS security credentials or permissions (but the creator of the pre-signed URL must have the necessary permissions to upload this object).

You can use the pre-signed URLs also with other AWS services.

When you create a pre-signed URL (manually or using the AWS SDKs), you must provide your security credentials, a client method, number of seconds the pre-signed URL is valid for, and other parameters. Some AWS services have a predefined expiration time for the pre-signed URLs, for example, 300 seconds. In this case, the provided with the request value is ignored.

This blog post gives a brief overview of the Signature Version 4 signing process. As an example, the Secrets Manager GetSecretValue method is executed using the AWS CLI. The sample Python code snippet generates a pre-signed URL and a ready-to-use curl command for the same method.

Overview of the Signature Version 4 Signing Process

AWS supports Signature Version 4 and Signature Version 2. All AWS services except the Amazon SimpleDB service support Signature Version 4.

Signature Version 4 is the process of adding authentication information to HTTP requests sent to AWS. The Signature Version 4 signing process consists of four main steps.

Figure showing four steps of the Signature Version 4 signing process.

You can observe the steps in the output of an AWS CLI command if you add --debug flag to this command. For example, when you get a Secrets Manager secret value:

aws secretsmanager get-secret-value --secret-id 'QA/Database' --region us-east-1 --debug

In the debug output of the executed AWS CLI command, you can see the values used in the signing process.

2019-10-28 20:10:57,633 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=GetSecretValue) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'secretsmanager.GetSecretValue', 'Content-Type': 'application/x-amz-json-1.1', 'User-Agent': 'aws-cli/1.16.266 Python/3.6.8 Linux/4.14.146-93.123.amzn1.x86_64 botocore/1.13.2'}, 'body': b'{"SecretId": "QA/Database"}', 'url': 'https://secretsmanager.us-east-1.amazonaws.com/', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7f06feb60c18>, 'has_streaming_input': False, 'auth_type': None}}

Also, you can see the main steps of the signing process:

  1. Create the canonical request.
    2019-10-28 20:10:57,633 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
    POST
    /
    
    content-type:application/x-amz-json-1.1
    host:secretsmanager.us-east-1.amazonaws.com
    x-amz-date:20191028T201057Z
    x-amz-target:secretsmanager.GetSecretValue
    
    content-type;host;x-amz-date;x-amz-target
    07c56590fcaba9db6bf7e11d57479ccdbb1bf2535411db9e3555170951a9d3a0
    
    The hash of the payload is included in the canonical request.
    echo -n '{"SecretId": "QA/Database"}' | sha256sum
    07c56590fcaba9db6bf7e11d57479ccdbb1bf2535411db9e3555170951a9d3a0
    
    • The hash function is typically SHA256, and the corresponding algorithm is AWS4-HMAC-SHA256.
  2. Create the string to sign using the canonical request from Step 1 and additional metadata.
    2019-10-28 20:10:57,633 - MainThread - botocore.auth - DEBUG - StringToSign:
    AWS4-HMAC-SHA256
    20191028T201057Z
    20191028/us-east-1/secretsmanager/aws4_request
    92b47568e86b18d63f2c173aec1b3b89e7764898eebf84e67bd0440e32682475
    
  3. Calculate the signature using a signing key and the string from Step 2. The signing key is derived from your AWS secret access key.
    2019-10-28 20:10:57,633 - MainThread - botocore.auth - DEBUG - Signature:
    9f320e4b339a4f9629fa986744b12c36bfd9e543e612b15720a1e3e22c3590ac
    
  4. Add the signing information to the request.
    2019-10-28 20:10:57,633 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://secretsmanager.us-east-1.amazonaws.com/, headers={'X-Amz-Target': b'secretsmanager.GetSecretValue', 'Content-Type': b'application/x-amz-json-1.1', 'User-Agent': b'aws-cli/1.16.266 Python/3.6.8 Linux/4.14.146-93.123.amzn1.x86_64 botocore/1.13.2', 'X-Amz-Date': b'20191028T201057Z', 'Content-Length': '27', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIA****************/20191028/us-east-1/secretsmanager/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-target, Signature=9f320e4b339a4f9629fa986744b12c36bfd9e543e612b15720a1e3e22c3590ac'}>
    
    The AWS CLI included in the HTTP request a header named Authorization and all headers used in the canonical request from Step 1. The Host header is added automatically by the Python requests library. The payload is also submitted with the request.
  • In this example, the long-term access key credentials (access key ID and secret access key) were used by the AWS CLI. In case of temporary access keys, in addition to the access key ID and secret access key, the temporary security credentials include a security token, and this token is also used in the Signature Version 4 signing process.

How to Manually Create a Pre-Signed URL

To create pre-signed URLs, you can use one of the AWS SDKs or do it manually with a few lines of source code. For example, the following Python code generates a pre-signed URL and a ready-to-use curl command for getting a Secrets Manager secret value. For further examples, refer to the AWS Documentation.

The code below is compatible with Python 2.7, Python 3.6, and Python 3.7. It does not have botocore, boto3, or any other third-party dependency.

In this code, the IAM Role credentials are used, so you must run it, for example, in a Lambda function or on an EC2 instance that has an IAM role attached. In the IAM role, you must grant permissions for the used Secrets Manager resource and action (in this example, for the QA/Database secret and the secretsmanager:GetSecretValue action).

The generated pre-signed URL is valid for 5 minutes. This is the current fixed value for the Secrets Manager service. If you provide a different value with the request, this value will be ignored.

Download get-secret-value-pre-signed-url.py.

#!/usr/bin/env python

# CloudBriefly

# The original source code is available at
# https://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html
#
# The following changes have been done:
# - Adjusted to the Secrets Manager API (GetSecretValue)
# - Added support of the IAM Role credentials
# - Added output to the console of a ready-to-use curl command
# - Changed coding style and optimized the code
#
# Copyright 2010-2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# This file is licensed under the Apache License, Version 2.0 (the "License").
# You may not use this file except in compliance with the License. A copy of the
# License is located at
#
# http://aws.amazon.com/apache2.0/

from __future__ import print_function
from datetime import datetime
import hashlib
import hmac
import json
import sys
if sys.version_info[0] > 2:
    from urllib.parse import quote_plus
    from urllib.request import urlopen
else:
    from urllib import quote_plus, urlopen

def sign(key, message):
    return hmac.new(key, message.encode('utf-8'), hashlib.sha256).digest()

def create_pre_signed_url():
    # Request values
    canonical_uri = '/'
    method = 'POST'
    service = 'secretsmanager'
    region = 'us-east-1'
    host = 'secretsmanager.%s.amazonaws.com' % region
    endpoint = 'https://%s' % host
    amz_expires = '300'
    request_parameters = '{"SecretId": "QA/Database"}'

    # HTTP headers
    content_type = 'application/x-amz-json-1.1'
    amz_target = 'secretsmanager.GetSecretValue'

    # Get the IAM Role credentials
    metadata_url = 'http://169.254.169.254/latest/meta-data/iam/security-credentials/'
    iam_role = urlopen(metadata_url).read().decode('utf-8')
    credentials = json.load(urlopen(metadata_url + iam_role))

    now = datetime.utcnow()
    amz_date = now.strftime('%Y%m%dT%H%M%SZ')
    date_stamp = now.strftime('%Y%m%d')

    headers = {'Content-Type': content_type, 'Host': host, 'X-Amz-Date': amz_date,
               'X-Amz-Target': amz_target}

    # Step 1: Create the canonical request
    credential_scope = '%s/%s/%s/aws4_request' % (date_stamp, region, service)
    canonical_headers = '\n'.join(h.lower() + ':' + headers[h] for h in sorted(headers))
    signed_headers = ';'.join(h.lower() for h in sorted(headers))
    canonical_querystring = ''.join([
        'X-Amz-Algorithm=AWS4-HMAC-SHA256',
        '&X-Amz-Credential=%s' % quote_plus(credentials['AccessKeyId'] + '/' +
                                            credential_scope),
        '&X-Amz-Date=%s' % amz_date,
        '&X-Amz-Expires=%s' % amz_expires,
        '&X-Amz-Security-Token=%s' % quote_plus(credentials['Token']),
        '&X-Amz-SignedHeaders=%s' % quote_plus(signed_headers)])
    payload_hash = hashlib.sha256(request_parameters.encode('utf-8')).hexdigest()

    canonical_request = '\n'.join([
        method, canonical_uri, canonical_querystring, canonical_headers, '',
        signed_headers, payload_hash])

    # Step 2: Create the string to sign
    string_to_sign = '\n'.join([
        'AWS4-HMAC-SHA256', amz_date, credential_scope,
        hashlib.sha256(canonical_request.encode('utf-8')).hexdigest()])

    # Step 3: Calculate the signature
    signing_key = \
        sign(sign(sign(sign(('AWS4' + credentials['SecretAccessKey']).encode('utf-8'),
                            date_stamp),
                       region),
                  service),
             'aws4_request')

    signature = hmac.new(signing_key, (string_to_sign).encode('utf-8'),
                         hashlib.sha256).hexdigest()

    # Step 4: Add the signing information to the request
    curl_command = "curl -X %s -H %s -d '%s' '%s/?%s&X-Amz-Signature=%s'" % (
        method, ' -H '.join("'" + h + ': ' + headers[h] + "'" for h in sorted(headers)),
        request_parameters, endpoint, canonical_querystring, signature
    )
    print(curl_command)

create_pre_signed_url()

The resulting curl command for the time 2019-10-28 20:10:57 UTC would look as follows:

curl -X POST -H 'Content-Type: application/x-amz-json-1.1' -H 'Host: secretsmanager.us-east-1.amazonaws.com' -H 'X-Amz-Date: 20191028T201057Z' -H 'X-Amz-Target: secretsmanager.GetSecretValue' -d '{"SecretId": "QA/Database"}' '<Pre-signed URL>'
  • To determine the request values for your use case, refer to the API Reference of the corresponding AWS service. Alternatively, you can investigate the debug output of the corresponding AWS CLI command (see Overview of the Signature Version 4 Signing Process).