Binary Responses from AWS Lambda via API Gateway

Published: at 09:32 PM

Recently, while working on an Inky Frame Display for TFL Bus and Weather Updates, I wanted to render a PNG image and return it directly as the body of the response from my AWS Lambda function. The documentation around this looked straightforward, but it was not as easy as it seemed. Several hours of research, and trial-and-error later, I managed to connect enough clues together in the right shape to get it working. Here’s a look at my journey, and a concise example which shows the minimum configuration required to achieve the same outcome.

Background

I like using AWS SAM to build and deploy Lambda functions. AWS SAM abstracts away a lot of boilerplate and chores involved in creating a Lambda function, while providing easy integrations with commonly used AWS services such as an API Gateway, or S3, or DynamoDB.

SAM takes away most of the repetitive work around building, deploying, and updating Lambda functions, and also provides easy access to the underlying CloudFormation, for situations where SAM does not have an easy abstraction available for required services or features.

For one of my fun personal projects, I wanted the ability to return an image from my Lambda function. I first looked up the documentation linked from my sam hello-world template.

The output format section of the documentation describes a key named isBase64Encoded:

If body is a binary blob, you can encode it as a Base64-encoded string by setting isBase64Encoded to true and configuring */* as a Binary Media Type. Otherwise, you can set it to false or leave it unspecified.

A note below links to a document which describes how to use the AWS console UI to manually configure an API Gateway to enable binary response support.

This could get me going, but where’s the fun in that? Having the ability to configure this once is great, but I wouldn’t enjoy having a manual step to perform every time I deployed a similar function. I have a strong preference for IaC (Infrastructure as Code) and repeatable outcomes, having many years ago learned about cattle vs. pets, and all that newly-fangled jazz.

I was looking for a way to achieve this in my code, so I would never have to remember a manual step, or spend another day sometime in the future, having to debug why it didn’t “just work”.

I tried setting the content-type in my response to image/png, and the isBase64Encoded key to true, just to see how the API gateway handled it.

My browser displayed a broken image:

I then used curl to see what the gateway was returning:

Terminal window
$ curl -D /dev/stderr -s https://b94vvwci85.execute-api.eu-west-2.amazonaws.com/Prod/png/ |hexdump -C|head


HTTP/2 200
content-type: image/png
content-length: 3668


00000000  69 56 42 4f 52 77 30 4b  47 67 6f 41 41 41 41 4e  |iVBORw0KGgoAAAAN|
00000010  53 55 68 45 55 67 41 41  41 5a 41 41 41 41 47 51  |SUhEUgAAAZAAAAGQ|
00000020  43 41 4d 41 41 41 43 33  59 63 62 2b 41 41 41 41  |CAMAAAC3Ycb+AAAA|
00000030  41 58 4e 53 52 30 49 42  32 63 6b 73 66 77 41 41  |AXNSR0IB2cksfwAA|
00000040  41 41 6c 77 53 46 6c 7a  41 41 41 4c 45 77 41 41  |AAlwSFlzAAALEwAA|
00000050  43 78 4d 42 41 4a 71 63  47 41 41 41 41 6b 39 51  |CxMBAJqcGAAAAk9Q|
00000060  54 46 52 46 2f 2f 2f 2f  38 50 44 77 41 41 41 41  |TFRF////8PDwAAAA|
00000070  46 68 59 57 64 58 56 31  50 7a 38 2f 36 65 6e 70  |FhYWdXV1Pz8/6enp|
00000080  31 4e 54 55 66 33 39 2f  32 4e 6a 59 63 6e 4a 79  |1NTUf39/2NjYcnJy|
00000090  70 36 65 6e 52 6b 5a 47  34 65 48 68 6d 35 75 62  |p6enRkZG4eHhm5ub|

It looked like the API Gateway was returning the correct content-type header, but it was sending my base64 encoded response as-is, without decoding it first. Changing the body key to the raw, unencoded binary image data resulted in an HTTP 500 error from the API Gateway.

I needed to do more research to discover a reliable, automatic way to correctly configure API Gateway.

Initial research

My requirement was not unique, perhaps a bit rare, but I was sure that someone else had solved it before, so I went to the search engines, hoping for a well-documented solution.

I found a few types of discussions and articles on this topic:

  • The vast majority of discussions were repeating what the AWS documentation said, and describing the manual way to configure API Gateways. Not what I was looking for!
  • A number of posts discussing how this can be achieved via other IaC frameworks, such as serverless. Probably works, but not relevant for my use case.
  • A few discussions where requiring the client the pass an accept header matching the response content-type header was the only viable solution. Not being a big fan of a tech limitation wagging the workflow dog, I made a note but kept looking for a better solution.
  • A couple of github bugs logged in the AWS SAM CLI GitHub repository: one remained open, but the other issue was closed, with people continuing to complain that it doesn’t actually work for them.
  • A few discussions suggesting to run an AWS CLI command which will configure the SAM-created API Gateway with the necessary changes. Not the worst, but not the ideal solution either.

Knowing that this was possible with other IaC frameworks, and also with manual configuration, either via the console or via AWS CLI, I strongly felt that a clean, straightforward way to configure this was possible. I chose to look more closely at the specific BinaryMediaTypes configuration in API Gateway.

Further exploration

My first approach was to look more closely at related issue discussions in the SAM CLI github repository. I found a configuration suggested by an AWS employee which seemed to have been accepted positively by the reporter, so I tried applying it to my test function. Initially, I had the same result: API Gateway was returning base64-encoded data, and not decoding it at all. But with new knowledge, I remembered a note from the isBase64Encoded example page:

To use a web browser to invoke an API with this example integration, set your API’s binary media types to */*. API Gateway uses the first Accept header from clients to determine if a response should return binary media. To return binary media when you can’t control the order of Accept header values, such as requests from a browser, set your API’s binary media types to */* (for all content types).

To test if this limitation was getting in my way, I tried sending an accept header to the gateway, and the response was exactly what I was originally looking for:

Terminal window
$ curl -H accept:image/png -D /dev/stderr -s https://b94vvwci85.execute-api.eu-west-2.amazonaws.com/Prod/png/ |hexdump -C|head


HTTP/2 200
content-type: image/png
content-length: 2749


00000000  89 50 4e 47 0d 0a 1a 0a  00 00 00 0d 49 48 44 52  |.PNG........IHDR|
00000010  00 00 01 90 00 00 01 90  08 03 00 00 00 b7 61 c6  |..............a.|
00000020  fe 00 00 00 01 73 52 47  42 01 d9 c9 2c 7f 00 00  |.....sRGB...,...|
00000030  00 09 70 48 59 73 00 00  0b 13 00 00 0b 13 01 00  |..pHYs..........|
00000040  9a 9c 18 00 00 02 4f 50  4c 54 45 ff ff ff f0 f0  |......OPLTE.....|
00000050  f0 00 00 00 16 16 16 75  75 75 3f 3f 3f e9 e9 e9  |.......uuu???...|
00000060  d4 d4 d4 7f 7f 7f d8 d8  d8 72 72 72 a7 a7 a7 46  |.........rrr...F|
00000070  46 46 e1 e1 e1 9b 9b 9b  b0 b0 b0 77 77 77 ae ae  |FF.........www..|
00000080  ae 30 30 30 ca ca ca 89  89 89 7e 7e 7e b4 b4 b4  |.000......~~~...|
00000090  85 85 85 ba ba ba 63 63  63 50 50 50 1c 1c 1c 2c  |......cccPPP...,|

Armed with this new assurance, I tried just changing the BinaryMediaTypes configuration to include */*, and it worked perfectly, without requiring specific accept headers to return the binary response.

Minimum implementation

I first used the sam hello-world template to create the Lambda function.

I then modified the app.py file to add special handling for a /png/ resource path, to return a base64 encoded file instead of a JSON response.

hello_world/app.py
import base64
import boto3
import json


# import requests




def lambda_handler(event, context):
    """Sample pure Lambda function


    Parameters
    ----------
    event: dict, required
        API Gateway Lambda Proxy Input Format


        Event doc: https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html#api-gateway-simple-proxy-for-lambda-input-format


    context: object, required
        Lambda Context runtime methods and attributes


        Context doc: https://docs.aws.amazon.com/lambda/latest/dg/python-context-object.html


    Returns
    ------
    API Gateway Lambda Proxy Output Format: dict


        Return doc: https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html
    """


    # try:
    #     ip = requests.get("http://checkip.amazonaws.com/")
    # except requests.RequestException as e:
    #     # Send some context about this error to Lambda Logs
    #     print(e)


    #     raise e


    if event['requestContext']['resourcePath'] == '/png':
        with open("hello.png", "rb") as image_file:
            image = base64.b64encode(image_file.read())
        response = {
            'headers': {"content-type": "image/png"},
            'statusCode': 200,
            'body': image,
            'isBase64Encoded': True
        }
    else:
        response = {
            'headers': {"Content-Type": "text/html"},
            'statusCode': 200,
            'body':  (f"<h1>hello world</h1>\n"
              "<img src='https://{event['requestContext']['domainName']}/"
              "{event['requestContext']['stage']}/png/' />"),
        }


    return response

After that, I modified the SAM template.yml file to add a custom API Gateway resource with the necessary BinaryMediaTypes configuration

template.yml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  aws-sam-lambda-gw-binary-response


  Sample SAM Template for aws-sam-lambda-gw-binary-response


# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 3
Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: hello_world/
      Handler: app.lambda_handler
      Runtime: python3.11
      Architectures:
        - x86_64
      Events:
        HelloWorld:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /hello
            Method: get
        HelloPNG:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /png
            Method: GET
            RestApiId: !Ref 'HelloWorldApi'
  HelloWorldApi:
    Type: AWS::Serverless::Api
    Properties:
      StageName: Prod
      BinaryMediaTypes:
        - "*/*"


Outputs:
  # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
  # Find out more about other implicit resources you can reference within SAM
  # https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
  HelloWorldApi:
    Description: "API Gateway endpoint URL for Prod stage for Hello World function"
    Value: !Sub "https://${HelloWorldApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"
  HelloWorldFunction:
    Description: "Hello World Lambda Function ARN"
    Value: !GetAtt HelloWorldFunction.Arn
  HelloWorldFunctionIamRole:
    Description: "Implicit IAM Role created for Hello World function"
    Value: !GetAtt HelloWorldFunctionRole.Arn

Defining a custom API allows you to separate binary response-producing handlers from handlers that may have proxy and template configurations applied. If a separate API resource is not needed, the same result can be achieved by applying the configuration globally:

template.yml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  aws-sam-lambda-gw-binary-response


  Sample SAM Template for aws-sam-lambda-gw-binary-response


# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 3
  Api:
    BinaryMediaTypes:
      - "*/*"
Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: hello_world/
      Handler: app.lambda_handler
      Runtime: python3.11
      Architectures:
        - x86_64
      Events:
        HelloWorld:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /hello
            Method: get
        HelloPNG:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /png
            Method: GET


Outputs:
  # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
  # Find out more about other implicit resources you can reference within SAM
  # https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
  HelloWorldApi:
    Description: "API Gateway endpoint URL for Prod stage for Hello World function"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"
  HelloWorldFunction:
    Description: "Hello World Lambda Function ARN"
    Value: !GetAtt HelloWorldFunction.Arn
  HelloWorldFunctionIamRole:
    Description: "Implicit IAM Role created for Hello World function"
    Value: !GetAtt HelloWorldFunctionRole.Arn

Conclusion

ex duris gloria

Previous Post
AWS re:Invent re:Cap London 2025
Next Post
TFL Bus and Weather Display - Part 1