rocket_launch The Challenge: GDAL in Serverless Golang
I wanted to try using GDAL in a Golang application with the Airbus godal library and deploy it to AWS Lambda as a Docker container. I had read that cold start times are significantly reduced when using AWS OS-based Docker containers because the caching of these layers is more effective.
I had an example for GDAL using an older Amazon Linux 2 Docker image, but I really wanted to compile
it myself to squeeze out the last milliseconds of speed. The problem was that the package manager
changed from yum to dnf,
so I could not simply copy the Docker YAML and replace the base Docker image.
In times before AI, I spent months working through countless nights to make it work, and I learned a great deal about compiling GDAL, its dependencies, and their interplay. I removed everything that was not needed for my use case to make the Docker image as small as possible, thereby speeding up cold start times.
check_circle The Result
The final working result is available on GitHub: al2023-gdal
This repository contains a fully functional GDAL compilation for Amazon Linux 2023, optimized for AWS Lambda deployments. It includes all the necessary build configurations and stripped-down dependencies to create minimal Docker images with fast cold start times.
Key Features
- • Custom GDAL compilation optimized for Amazon Linux 2023
- • Compatible with AWS Lambda Docker containers for better layer caching
- • Minimal dependencies for reduced image size and faster cold starts
- • Works seamlessly with the Airbus godal library for Golang
school What I Learned
This project was a deep dive into the internals of geospatial software compilation. Through trial and error over many months, I gained hands-on expertise in:
GDAL Dependencies
Understanding the complex web of GDAL's dependencies and how to compile them from source.
Build Optimization
Learning which dependencies are essential and which can be removed for specific use cases.
Amazon Linux Migration
Navigating the changes from Amazon Linux 2 to 2023, including the yum to dnf transition.
Docker Optimization
Techniques for minimizing Docker image size and improving Lambda cold start performance.