Transforming payload with VTL in API Gateway — EventBridge integration

By Matej Vasko

This article assumes you already have a working integration between AWS API Gateway and Amazon EventBridge.

It seems like you’re really trying to avoid using another lambda to handle the payload transformation. Good choice*. The number of lambdas in a serverless infrastructure can quickly get out-of-hand and you should avoid creating unnecessary resources whenever it’s possible.

Amazon REST API Gateway offers a solution in the form of Mapping templates. The basic passthrough integration looks as follows:

{
  "Entries" : [
    {
      "DetailType": "foo type",
      "Source": "com.company",
      "EventBusName": "event-bus-name",
      "Detail": "$util.escapeJavaScript($input.json('$'))"
    }
  ]
}

• DetailType: Identifies, in combination with the Source field, the fields and values that appear in the Detail field.
• Source: Identifies the service that generated the event.
• EventBusName: The name of the targeted event bus.
• Detail: The payload of the event (this field is where it gets tricky).

What do we want to achieve?

Imagine that some third party is sending unstructured, strange looking data to your API.

{
    "eventType": "ItemSold",
    "firstName_TXT": "Gordon",
    "lastName_TXT": "Ryan",
    "AddressLine": "1 Palmer Ave",
    "City": "San Juan",
    "Country": "Puerto Rico",
    "PurchasedItem_TEXT": "Surfing Board",
    "PurchasedAt_DATE": "2021-01-30T08:30:00Z",
    "BDAS_4333": "WVR8tHJsfF"
}

Before the payload hits the EventBridge, you want it to look like this:

{
    "event": "ITEM_PURCHASED",
    "customer": {
        "firstName": "Gordon",
        "lastName: "Ryan"
    },
    "address": {
        "line": "1 Palmer Ave",
        "city": "San Juan",
        "country": "Puerto Rico"
    },
    "purchase": {
        "item": "Surfing Board",
        "date": "2021-01-30T08:30:00Z"
    }
}

Nice and clean, without the BDAS_4333 field that you don’t even need.

The transformation

Folks at AWS have decided that you’d use VTL to handle the transformation. The whole stumbling-block is that EventBridge expects the value of Detail field as a stringified, escaped JSON.

{
    ...
    "Detail": "{\"firstName\":\"Gordon\"}"
}

For such an easy transformation you could write the Mapping template inline:

{
  "Entries" : [
    {
      "DetailType": "foo type",
      "Source": "com.company",
      "EventBusName": "event-bus-name",
      "Detail": "{\"firstName\": "$util.escapeJavaScript($input.json('$.firstName_TXT'))}"
    }
  ]
}

To handle the transformation of our example, the Mapping template can look as such:

#set($inputRoot = $input.path('$'))
#set($eventMap = {
    'ItemSold' : 'ITEM_PURCHASED',
    'ItemReturned' : 'ITEM_RETURNED'
})
## Event
#set($event = '"' + $eventMap.get($inputRoot.eventType) + '"')
## Customer
#set($firstName = '"' + $inputRoot.firstName_TXT + '"')
#set($lastName = '"' + $inputRoot.lastName_TXT + '"')
## Address
#set($line = '"' + $inputRoot.AddressLine + '"')
#set($city = '"' + $inputRoot.City + '"')
#set($country = '"' + $inputRoot.Country + '"')
## Purchase
#set($item = '"' + $inputRoot.PurchasedItem_TEXT + '"')
#set($date = '"' + $inputRoot.PurchasedAt_DATE + '"')
#set($customerMap = {
    '"firstName"' : $firstName,
    '"lastName"' : $lastName
})
#set($addressMap = {
    '"line"' : $line,
    '"city"' : $city,
    '"country"' : $country
})
#set($purchaseMap = {
    '"item"' : $item,
    '"date"' : $date
})
#set($payloadMap = {
    '"event"' : $event,
    '"customer"' : $customerMap,
    '"address"' : $addressMap,
    '"purchase"': $purchaseMap
})
#set($payload= $payloadMap.toString().replace("=", ":"))
{
    "Entries" : [
        {
            "DetailType": "foo type",
            "Source": "com.company",
            "EventBusName": "event-bus-name",
            "Detail": "$util.escapeJavaScript("$payload")"
        }
    ]
}

Whoa, that’s a big jump. Let’s break it down.

1. First we save the incoming payload in $inputRoot variable.
2. We map the incoming $inputRoot.eventType to one of our desired values (ITEM_PURCHASED or ITEM_RETURNED) and save it in $event variable.
3. Extract all the desired fields and wrap them in double quotes.
4. Construct $customerMap, $addressMap and $purchaseMap maps.
5. Construct the $payloadMap.
6. Finally we stringify the $payloadMap. Since VTL map uses “=” instead of “:” to separate key-value pairs we’ll need to call .replace(“=”, “:”) on the stringified map.

Conclusion

As you’ve probably noticed by now, working with VTL Mapping template is no picnic. Extracting the variables and wrapping them in double quotes seems messy, but it was the only approach that kept the code somehow structured. Make sure that your VTL syntax is on point. Leaving one trailing comma inside map would break the whole thing. Or save yourself some time and use a lambda instead.