Fronting a Third Party API with AWS Lambda and Pydantic

By Jasper Wagen

In microservice architecture it’s sometimes the case that multiple services within your domain need to hit the same third-party API. Building clients for this API into each of your services leads to code duplication and makes swapping out one API provider for another much more difficult.

A common solution to this is to front the third-party API with your own internal API which handles the request logic, transformation and validation of the data. This post will go through a short example of how using Python with Pydantic, coupled with AWS Lambda and API Gateway is an effective and easy solution to this problem.

But what is Pydantic?

In brief, Pydantic is a python library that enforces type hints and provides validation tools through the use of data models. This is useful in an API request context because it ensures request data is as you expect and throws errors if not. For more information about Pydantic and its uses see here.

Dog Walking Site

Let’s say that you’ve been commissioned to build a dog walking site where dog owners can find people to walk their dogs. This site consumes the national dog registry which provides information used in both account creation and a separate special offer system.

One way to tackle this is to front the Dog Registry API with your own API, the infrastructure for which is shown below.

The GetDog lambda handles hitting the Dog Registry and then transforming the returned json into the business model required by your services using Pydantic.

I/O

First let’s look at the expected input and output of the lambda handler.

Input:

Output:

Off the bat there’s a few differences between the input and output we’ll have to cater for. The input is nested and has more fields than the output. Furthermore, the input is camel cased whereas the output requires snake case and we don’t need the age field in the output business model. These transformations can be neatly handled in Pydantic.

Models

The first step is to model the inputs and outputs as classes that extend Pydantic’s BaseClass, shown below. This snippet demonstrates a couple of nice features of Pydantic.

Nested Models

You can see from the DogRegistryModel (input) that nested models are simply handled by specifying the type of a key as another model. In this case owner is given the type Owner, a class defined at the top of the file.

Optional

Additionally we’ve made the owner optional (for the sad case of stray dogs), this can be achieved using python’s built in typing.Optional and placing the type in square brackets. We can then set a default if the value is not specified after the ‘=’, in this case set to None.

Key Change

One of the requirements was to change the case of our keys from camel case to snake case. Key name changes are handled neatly in by setting an alias, Field(alias=”dateOfBirth”). To read more about Field options see here.

Type Coercion

By default Pydantic will try to coerce the provided value to the type specified. For example if the hight field is supplied and integer, v, Pydantic will call float(v). This behaviour can be disabled using strict types. For more about type coercion see here.

Validation

In addition to typing, Pydantic can set custom validators for fields. In our example it would be good to validate that each dog’s date of birth obeys the laws of causality, and they weren’t (aren’t going to be?) born in the future.

Before the date_of_birth field can be validated it needs to be changed to the date type. This can be done by simply specifying the type as datetime.date and relying on type coercion to convert from the provided string.

To validate the field we can use the @validator decorator, which takes a key name as an argument. We can then simply pass date_of_birth to the method and check if it is before or after today, raising a ValueError if it is in the future else returning its value.

Output Model

So with the input model sorted we need to create the required output business model that we will be using in our domain.

Defining this model is pretty simple and just relies on a few of the concepts explained above. One issue you may have spotted is that owner in our output model is required to be the owner.name field in the input model, this is where a transform method comes in handy.

Transforming Models

There are a few ways to transform one model into another in Pydantic but the one I like and we’ll be using for this example is a transform method.

The transform method sits in the DogRegistryModel class and when called simply feeds selected arguments into the fields on the DogUserModel. You can see how this allows for flexibility when transforming between models, meeting our requirement of reducing the owner field down to just the owners name.

That’s all we need to do on the models, so now the final part is writing the lambda handler.

Handler

With all of the heavy lifting done in the Pydantic models, the lambda handler becomes incredibly lightweight.

First we take the event, grab the name of the dog we want to search, and fire a get request. We break the response into the DogRegistryModel and transform it into the DogUserModel. Finally we return the model as json by dumping the dict output.

Wrapping Up

The code we’ve gone through here could then be deployed to AWS Lambda with a IAS service of your choice. If you’d like to learn how to do this check out this guide on AWS Sam and perhaps this one on API Gateway Lambda integration.