I’ve been working recently on a personal project to investigate how much of the Overpass query language could be implemented using a different back end database, namely the QLever database.

Overpass is the most widely used public interface for querying OSM data. It is relatively easy to use and understand and its outputs in GeoJSON or OSM XML are readily integrated into OSM or other geospatial tool chains for visualization or other processing.

But Overpass is somewhat of a victim of its success. Demand for the public Overpass service has recently outpaced server capacity. My earlier work on a container image for Overpass was intended to help scale capacity by making it easier for data consumers to run their own local Overpass servers.

This project looks in a different direction. What if the Overpass QL with its ease of use and its integration into other tools were available on top of a different data source?

QLever

QLever is a SPARQL database developed by the Chair for Algorithms and Data Structures at the University of Freiburg. SPARQL is an RDF query language where the data are represented as “triples” of a subject, predicate, and object.

QLever represents geospatial data as WKT and can perform geospatial operations on WKT. The osm2rdf conversion generates WKT data for every OSM element. So, the data in QLever stores the complete geometry from OSM.

But the key to QLever’s performance with OSM data is that in the conversion from OSM PBF to TTL (Terse RDF Triple Language), osm2rdf generates triples for every element to describe their spatial relations with other elements. That is, osm2rdf pre-computes the spatial relations sfIntersects, sfContains, sfCovers, sfTouches, sfCrosses, sfOverlaps, and sfEquals between all OSM elements.

… See full entry

I’d like to announce the release of a new Docker container instance for Overpass.

I’ve been running Overpass locally for a couple of years and have dealt with some reliability issues by improving the shell scripts that drive database updates.

I’ve been sharing those shell scripts with people but I wanted a better way to distribute them. So, I put some effort into cleaning up the scripts (with some significant improvements – see below!) and packaged them in a container build.

Another Overpass Container?

Yes!

There’s already the excellent wiktorn/overpass-api which does a great job of setting up a local Overpass instance. If that looks good to you, you should definitely use it!

For my own use, I wanted something with all the improvements to stability and reliability that I had rolled into the shell scripts, but that would stay true to the original Overpass source code and that would expose all the same configuration options as if it were running on bare metal. I also wanted all the operational support that I already had locally, like configurable area generation intervals and periodic database backups.

Then, given the issues with the performance of public Overpass servers, and the efforts to limit large-scale usage, it seems that having more Overpass servers can only help the community.

What’s Different About This Container?

Better Database Updates

First, the database update process is much more robust. When things go wrong during Overpass database updates, the database files can be corrupted. And when that happens you have to start over with a fresh database, either from a backup or from a fresh download. That can mean many hours of downtime.

The shell scripts for database updates have been improved to shut down as gracefully under controlled circumstances and to handle uncontrolled shutdowns as safely as possible.

… See full entry

Many OSM validators will report an error if a waterway crosses a highway without some additional tags or structure at the intersection, such as a bridge, culvert, or ford. That makes a lot of sense if you’re working in an area where the roads and waterways are already well mapped and where the waterways are actually wet. Those assumptions might work well in Europe, for example. They don’t work well in the Southwest United States.

Of the possible resolutions, adding a node with ford=yes is the easiest, so many mappers will do this to satisfy the validator – sometimes without looking closely at the situation. Then you get results like this, where the ford=yes nodes are meaningless.

If you get a warning from a validator that a waterway and highway intersect and need additional tagging, there are some things to consider first:

… See full entry

This diary entry is also available as a personal page on the OSM wiki.

Some History of GNIS Imports

GNIS is a database developed by USGS that contains information about the official, standardized names for geographic features in the US.

From 2008 to 2011, there was an effort to import basic data into the US map, including records from GNIS and records from TIGER and NHD that are cross-referenced to GNIS. You can see some of the history of these imports based on the tags they used.

The gnis:feature_id tag was typically used for imports of many types of GNIS records. The gnis:id tag was used for imports of nodes for Populated Places, i.e. place=city, town, village, hamlet, etc. The tiger:PLACENS tag was used for imports of civil boundaries from the US Census Bureau datasets and contained a GNIS Feature ID value. The NHD:GNIS_ID tag was used for imports of waterways and other hydrographic features from the National Hydrography Dataset and also contained a GNIS Feature ID value.

… See full entry

In the course of working with GNIS data from the US Geological Survey, I’ve sometimes been frustrated with the limited range of expression in OSM tags for natural features. For example, we have a lot of tags that can be applied to a Bench as a place for a people to sit, but nothing specific to identify a Bench as a geographic landform other than tagging the edges as natural=cliff or natural=earth_bank.

There have been some good efforts to improve geological tagging, such as the Proposal for additional volcanic features and the Categories of Sea Areas, which give us broader vocabularies for some features. Strangely, the seamark:sea_area:category=* tag set is more expressive for undersea features than the OSM tags we have for features on land!

So, I decided to put together a Glossary of landforms for OSM, based on a similar glossary on Wikipedia. In the process, I’ve found that OSM does have a broad set of tags for geographic features, although many of them have limited or no documentation.

I also think that there is an opportunity to expand the values of the geological=* tag to include more types of geological features. If the main tag for a features is natural=* or something similar, that can identify the general shape of the landform and be the main tag used by renderers. The addition of a geological=* tag can add more specificity to the feature and identify the nature and structure of the landform. For example, the famous Sugarloaf Mountain in Rio de Janeiro is not just a mountain, but a Bornhardt. So, we might consider adding a geological=bornhardt tag to the feature.

If you have an interest in mapping natural features, check it out:

Glossary of landforms

It’s certainly a work in progress and there are some prospective tags on the list that aren’t currently in use, but I hope it might be useful. If anyone has input, I’d be very happy to have some additional contributions to the effort!

The GNIS matching project I’ve been working on uses a lot of Overpass queries to find things in OSM. At some point during the project, I needed a faster, more reliable Overpass server than the public servers. So I built a local Overpass server as cheaply as I could. It’s working well. This is how you can build one for yourself.

Update

Things have changed a lot in the last couple of years. If you’re just arriving at this diary entry, you might check out the newer container build for Overpass. Otherwise, keep reading …

Why Would I Build My Own Overpass Server?

If you’re using the Overpass API for software development, you’re going to be running a lot of queries. You could use a public Overpass instance, but it’s more polite and a lot more efficient to run one locally. Also, public overpass servers have query limits that you may not like. And sometimes they go down or flake out, and then there’s nothing you can do but wait until the operators fix them. If you run your own server, your fate is in your own hands!

For most use cases, a cheap local Overpass server can be significantly faster than using one of the public Overpass servers. The setup described here is a lot smaller with a lot less computing power than those big public servers. But it doesn’t have the entire world hammering on it constantly. Also, Overpass queries can return huge amounts of data. The network latency and throughput is a lot better on your own local network segment than if you’re downloading results from halfway across the world.

I’d like to give a special thanks to Kumi Systems for hosting the public Overpass server that I abused until I set up my own server. They’re providing a great service for the OSM community!

Do I Really Want to Do This?

… See full entry

I’ve started a new project working with watmildon. While we were working together on applying the USGS Sq___ name changes to OSM we noticed was that there were often features in OSM that were out of sync with official name changes that happened years ago.

That got us thinking about walking through the USGS GNIS data set to find places where names had changed and OSM could be updated. After all, there are many features in OSM that have gnis:feature_id (and similar) tags that can be directly matched back to the GNIS data set.

After kicking the idea around for a while, we recently started writing some code. I’ve been working on a matching engine in C# that matches records from GNIS to OSM by Feature ID. The code also looks for likely matches where the feature name, primary tags, and geometry are close to the information from GNIS. So far the results are pretty good, but we’re still working on improving the matching.

Meanwhile, watmildon did some large scale statistical analysis on a local PBF file to look at the scale and scope of the problem. The results were very interesting!

Of the 2.3 million features in GNIS, there are only 1 million corresponding features with GNIS IDs in OSM. Some portion of these are surely existing features that just don’t have the gnis:feature_id (or similar) tags. But given our manual review of results from the matching code, there are a lot of GNIS features that are not present in OSM at all.

That’s not too much of a surprise. Some of the most common types of missing features are Streams, Valleys, Lakes, Springs, and Ridges – all things that not widely mapped in the US.

… See full entry

I’ve run across a few places where there seems to be some disagreement and confusion about how to distinguish between roads that should be tagged as highway=service versus highway=track. I see quite a few ways that get switched back and forth between the two tags each time a different mapper touches them.

So I figured I’d write up how I make the distinction. I understand that other mappers might think about these things differently, but here’s how I think about the two types of roads.

Service roads (highway=service) are:

• used to provide motor vehicle access from a through road or a local road to a specific destination (building, etc.)
• typically very short
• typically used for a single purpose
• often one lane (although sometimes wider)
• typically not named or numbered (i.e. no name or ref tags)

Some examples of highway=service: a driveway, an urban alley, a parking aisle, a short access road for utility equipment, an access road leading to one or more campsites, or an access road in a municipal dump.

Track roads (highway=track) are:

• local roads that are only wide enough for a single four-wheeled vehicle (i.e. dual-track on the ground)
• can be short or long (i.e. many miles)
• typically used for multiple purposes
• typically named and/or numbered when they are approved public routes of travel

Some examples of highway=track: a dual-track dirt road, a remote single-lane paved road, a graded single-lane road along a canal or railway, or a dirt road along the path of a power line.

The place that seems to cause the most confusion is where a longer road is used to access some sort of infrastructure. For me, this is typically a highway=track. Although roads like this can be used to access infrastructure (e.g. towers for high-voltage power lines) they can also be used for through travel, recreation, and other purposes. Roads like this are often designated for multiple uses by the land manager responsible for them.

… See full entry

So, there’s this gem:

This is the Warren H. Brock Reservoir overlayed with the BLM 356 track from the latest BLM GTLF data set. Construction started on the reservoir in 2008, so the BLM GTLF data in this area is at least that old.

Sometimes dealing with external data sources requires a little creative interpretation. Aerial imagery shows a track that goes around the reservoir, so that’s the new alignment for BLM 356.

While I’ve been working on BLM Ground Transportation Linear Features (i.e. highways) in Imperial County, I took a small diversion to put in the BLM Off-Highway Vehicle Areas in California. Four of the major BLM OHV areas are in Imperial County, so it was relevant. These boundaries are important because many of the OHV areas are “open,” allowing cross-country travel off of designated roads and trails.

I’ve been working with the BLM CA Off Highway Vehicle Designations data set, which has 31 OHV areas in California and one OHV area from Nevada that slipped in because it’s managed by a BLM field office in California.

As I started adding the OHV areas, I noticed that almost all of these areas have never been mapped. Some of these areas are notable institutions in the off-road community, like Imperial Dunes (aka Glamis) and Johnson Valley (home of King of the Hammers). So adding these areas is a significant contribution to the map.

Tagging these areas is a little bit of a challenge. After some discussion with Minh Nguyen, I settled on landuse=recreation_ground and leisure=offroad_driving to tag all the OHV areas. The recreation_ground tag is a slightly odd fit but it seems close enough to be appropriate. And some renderers have an idea what to do with it.

Some of the OHV areas only permit vehicles on designated routes. That’s no problem because the access tags go on highway features and the OHV area doesn’t need any additional tagging. But many OHV areas permit open cross-country riding, so tagging vehicle access is an issue for these areas. I settled on adding motor_vehicle and ohv access tags directly to the areas, with values like yes for unrestricted access, permit where vehicles must have a pass, and permissive for the shared-use area of Johnson Valley which is periodically closed for military use (but not on a predetermined schedule).

… See full entry

Many of the features we had to deal with in the Sq___ renaming were small streams and creeks. For some reason, Sq___ Creek seemed to be a very popular name. In most cases, the creeks weren’t present on OSM, and as part of the renaming, we decided to add missing features with the new names so that future mappers wouldn’t unknowingly add the features with the old Sq___ names. So, we had to map a lot of small creeks.

One of the challenges with mapping named waterways is identifying the full extent of the waterway. Where is the mouth of the waterway? Where is the source? Of the many branches upstream from the mouth, which branch is the identified course?

If you’re just working with GNIS, the GNIS data has two or sometimes three sets of coordinates for waterway features. The first coordinate is the mouth. If there is a second coordinate, it falls somewhere in the middle of the waterway. The last coordinate is the source.

From there, you can use topo and aerial maps to trace the course of the waterway by hand.

Alternatively, you can download the local data file from the National Hydrography Dataset, find the waterway you want, merge it into an OSM layer, and clean it up before uploading it. Here’s how that works:

… See full entry

I guess I didn’t keep up with this diary. I’ve been busy since that first entry.

I’ve gone through both Cleveland National Forest and San Bernardino National Forest to fill in and tag forest roads and trails. At this point, both forests should have all of the official routes of travel with their official names and refs. There will be some differences between OSM and the FS Topo data source where conditions on the ground are different from what USFS has in their data set.

I also worked with a team of mappers to update all the Sq___ names that were changed by BGN/USGS in both OSM and Wikidata. (See https://www.doi.gov/pressreleases/interior-department-completes-removal-sq-federal-use and https://edits.nationalmap.gov/apps/gaz-domestic/public/all-official-sq-names.) That made me really happy because the unfortunately named local “Squaw Tit” that I used in my backcountry navigation course is now appropriately named “Mat Kwa’Kurr.”

I have two things going on now. I’m filling in the roads that CBP has made in the Jacumba Wilderness Area. There’s been a lot of activity out there recently and some construction work on two new border wall segments. This activity is controversial for a number of reasons, so I think it’s important to document the impact CBP has had in the area.

I’ve also started working with the BLM Ground Transportation Linear Features data set to fill in backcountry routes in SoCal.

I’m starting with the area around Superstition Mountain in Imperial County because it’s a popular area for off-road activities. The terrain there makes navigation challenging, so good maps can make a huge difference.

Unfortunately, the shifting sand dunes in the area also make mapping difficult. And the BLM data is not very good. The BLM tracks wander off into places where there are no tracks on the ground. And the BLM tracks take some improbably dangerous routes over surface features, like crossing the sharp crest of a soft sand dune.

… See full entry

I’m no stranger to GIS, so I jumped right in. Filled in BLM routes around Painted Gorge, Mica Gem, and Buck Canyon. Cleaned up some routes at Valley of the Moon and along Grapevine Canyon.

Still need to add a couple routes to Painted Gorge but I want GPS data first. There are discrepancies between BLM maps and aerial photos that can only be resolved by going there in person.

The Yuha Basin is also on my to do list, but I definitely want GPS data for that first. The BLM map isn’t that great and there are a lot of tracks on the ground that differ from the map.