OpenStreetMap
I’d like to announce the release of a new Docker container instance for Overpass.
I’ve been running Overpass locally for a couple of years and have dealt with some reliability issues by improving the shell scripts that drive database updates.
I’ve been sharing those shell scripts with people but I wanted a better way to distribute them. So, I put some effort into cleaning up the scripts (with some significant improvements – see below!) and packaged them in a container build.
Another Overpass Container?
Yes!
There’s already the excellent wiktorn/overpass-api which does a great job of setting up a local Overpass instance. If that looks good to you, you should definitely use it!
For my own use, I wanted something with all the improvements to stability and reliability that I had rolled into the shell scripts, but that would stay true to the original Overpass source code and that would expose all the same configuration options as if it were running on bare metal. I also wanted all the operational support that I already had locally, like configurable area generation intervals and periodic database backups.
Then, given the issues with the performance of public Overpass servers, and the efforts to limit large-scale usage, it seems that having more Overpass servers can only help the community.
What’s Different About This Container?
Better Database Updates
First, the database update process is much more robust. When things go wrong during Overpass database updates, the database files can be corrupted. And when that happens you have to start over with a fresh database, either from a backup or from a fresh download. That can mean many hours of downtime.
The shell scripts for database updates have been improved to shut down as gracefully under controlled circumstances and to handle uncontrolled shutdowns as safely as possible.
There are several other usability improvements to these scripts. They are much faster at downloading and updating the database when it’s catching up to the replication source. That substantially cuts down the time it takes to sync with the replication source.
The database update scripts are also much better at tracking the replication source to reduce latency. If you’re using Overpass while you’re editing, your changesets will show up in the database in a little more than a minute, so you can see the changes almost immediately.
Full Support for Minutely and Hourly Replication
The original Overpass source was only configured for minutely replication. If you wanted to use Overpass with an hourly or daily replication source (like Geofabrik), you either had to heavily edit the shell scripts or deal with some quirks and a lot of noisy, unnecessary polling.
The new scripts automatically detect the update frequency of the replication source and only check for new updates when they’re scheduled to be available. That reduces latency to download and process the updates, and reduces the load on replication servers from wasted polling.
Full Support for Planet and Extract Files
There are three ways you can set up a starting Overpass database. You can download a clone of an existing database, which was the only fully supported option with the original Overpass source, or you can download a planet file or a regional extract and import the data.
The new scripts cover initialization from a planet file or extract file and support both .osm.bz2 and .pbf source files. The download process is much faster, with parallel downloads over HTTP and support for torrent downloads.
And if you’re just cloning an existing database, that download now runs faster too!
Operational Improvements
The new scripts include an automated backup script that pauses database updates and copies the database files without stopping the query engine. That makes it easier to keep a local backup of the database for recovery in case something goes wrong.
Both the database directory and the backup directory can be bind mounted into the container so that the data is preserved across container runs.
There’s also a script to automatically clean out old replication data, so that it doesn’t fill your file system.
And the container includes automatic rotation of all the Overpass log files and process output files so they don’t fill your file system either.
The container also keeps tabs on the health of all the Overpass components. And it includes both liveness and health checks to fit in nicely with container orchestration.
Preconfigured with Nginx and Tuned for Performance
The container image includes nginx and fcgiwrap with tuning parameters that automatically account for the number of available CPUs and available container memory.
Unlike other Overpass instances, the default nginx configuration for this container build will return HTTP 429 responses to trigger client rate limiting if the server is busy. That means all the server resources are dedicated to processing queries that are already in progress, and additional queuing doesn’t slow down query processing.
Exposes Every Configuration Option
All the parameters used to configure Overpass can be overridden from their defaults by setting environment variables. So, you can tweak whatever you like from your container’s .env file.
The improved scripts also expose their configuration options as environment variables. So, you can adjust the behavior of the new scripts without having to edit the code.
Tracks Latest Overpass Releases
The container build process uses CI to track the latest Overpass releases. The current container image is built on v0.7.62.11, and new container builds will track new releases as they come out.
Still Runs on Bare Metal
If you prefer not to run Overpass in a Docker container, all the improved scripts will still run on bare metal. Just clone the source from the GitHub repository and follow the instructions there to build a native instance.
Try It Out!
If you’re frustrated with overloaded public Overpass instances or public instances that don’t keep up with replication updates, the new container image gives you the option to run local Overpass server for your own use.
If you have an application that generates a lot of Overpass query traffic, consider setting up a private server to shift load away from the public servers.
And if you’re running a public Overpass server that has reliability issues, consider trying this container build as a way to improve long-term stability and reduce replication latency.
However it might be used, I hope this new container image will help the community by providing more options for Overpass query processing!
Docker Hub: https://hub.docker.com/r/b1tw153/overpass-api
Discussion