OpenStreetMap
There are a lot of peaks mapped in OpenStreetMap, ranging from proper mountains to mere pimples. They’re usually rendered the same regardless of how high they are:
However, with a bit of preprocessing and a bit of help it’s possible to render higher ones at a higher zoom level:
I was prompted to write this because of a few discussions over the last couple of months - people talking about the default tags on various OSM elements, reactions to Amazon mappers’ additions of “last mile” service roads and discussions about how to tag England and Wales’ public rights of way network.
This is very much a personal view and one that’s very “England and Wales” centric - there are lots of different approaches to rural tagging in general and footpath tagging in particular, and many places around the world (enlightened places like Scotland and Scandinavia) have different default access rights to the curious set we have ended up with in England and Wales.
Let’s start with an example - what’s the reasoning for the tags on this way in OSM?
(following on from this diary entry)
Supposing you would like to overlay a set of boundaries over the top of any existing map style - what would you need to do?
Using this as a reference, create a database to hold the new boundary data.
If a historical data file is what you want, download the data for the desired date and rename it to a file name that the “update_boundaries” script will recognise:
-rw-rw-r-- 1 renderaccount renderaccount 459723464 Nov 2 01:46 ukraine_2018-11-01T21:14:02Z.osm.pbf
You can also make other changes to this data if you wish using tools like osmium, osmfilter, osmosis, etc.
Modify this script to process your data.
Run the script to load the data that you have already downloaded:
sudo update_boundaries.sh current
Create a version of “mapnik.xml” (or whatever style file you are using) that looks in the database that you’ve loaded your boundary data into. In this case I just changed “dbname” references from “gis” to “gis3”.
Ensure that “/usr/local/etc/renderd.conf” knows about the new tile layer. The default one I use just has “ajt” in it; I created a copy of the last section, called it “ajt3” and changed URI and XML values.
[ajt3]
URI=/hot3/
TILEDIR=/var/lib/mod_tile
XML=/home/renderaccount/src/openstreetmap-carto/mapnik3.xml
HOST=localhost
TILESIZE=256
MAXZOOM=20
Restart renderd and apache2, and your new tiles will appear. Then in a simple Leaflet map add an overlay layer for your new tiles. This is an example that shows Leaflet overlays but there are many, many others, including on the main Leaflet site itself.
Here’s what it looks like:
When I developed the map style that you can see here from OSM-carto one thing that I didn’t carry forward was the representation of borders. Partly this was because it’s not easy to separate maritime ones from non-maritime ones, the the purple maritime country borders frankly don’t look very nice:
Partly it was also due to “clutter” - I didn’t want an arbitary boundary to get in the way of a river or stream or other feature.
Sometimes however, it would be useful to see what borders are where, so why not do it as an overlay? This turned out to be rather easier than anticipated.
I’ve been wanting to write an explainer for this for a while (even before this diary comment). The problem was that I wasn’t aware of a Docker example that (a) was reasonably up to date and (b) was designed around tile serving (rather than style design for example).
However, this docker example was recently mentioned by its author on IRC. It’s based on the building a tile server for 18.04 instructions and also handles importing data and running the tile server afterwards.
First things first, I installed Centos 7 on a virtual machine. I got a “minimal” iso image for that from here. I installed it (which involved turning the network on inside the installer and selecting the disk to install on). I also set a root password and added a non-root user (I’ll use the example “renderaccount” below), and noted the IP address that it obtained. I did a “yum update” to install any available software updates.
There are a few sets of instructions for “installing Docker on Centos” - DigitalOcean have one, and there’s one here which also contains lots of other useful information. Following the instructions there:
su -
yum install yum-utils device-mapper-persistent-data lvm2 wget
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install docker-ce
systemctl start docker
If you then do this:
systemctl status docker
In the output you should see “active (running)”. Do this:
systemctl enable docker
in order to have docker start every time and then exit from root and do this from you’re non-root user:
sudo usermod -aG docker $USER
So that you can control docker from that non-root user. Log out and back in and do this:
docker container run hello-world
It should say “Hello from Docker!” to show that everything is working OK. If instead you see “permission denied” it probably means you haven’t logged out and back in again.
I’m in the process of changing map.atownsend.org.uk so that zoom level 13 can still be used as a “route planning” layer but walls, hedges and ditches don’t get in the way. This wasn’t originally a problem, but people have since mapped lots of fences, hedges,walls and ditches, and it can be difficult to see footpaths at that zoom level now. Here’s a screen showing half new tiles, half old ones:
That corresponds to here on the site and here in OSM. The new version will probably be rolled out in a week or so’s time. Other zoom levels will be unchanged - you’ll still see hedges and ditches a couple of zoom levels earlier than with OSM’s “standard” style, and paths will still be legible 2 or 3 zoom levels earlier.
I’ve added a new page to the switch2osm guide: “Manually building a tile server (18.04 LTS)”.
Although Ubuntu 18.04 (“Bionic Beaver”) isn’t released yet, it’s due out fairly soon and daily builds can be downloaded from here. There are actually very few changes from the 16.04 version - mostly just updated versions of software (including some Mapnik fixes). Following these instructions shouldn’t take more than a couple of hours for small areas - the longest period of the setup is waiting for the shapefiles used by the style to download.
As before, the page is designed to be “the least you need to do” to get a rendering server working. As before I also wrote a wiki page which goes into a bit more detail, including other things that you might want to do.
It’s also worth mentioning than there are many more resources available now than there were a couple of years ago - see for example Ircama’s tutorials, and also this guide that covers the setup of an OSM Carto renderer within Docker.
(written in response to the discussions here and elsewhere, prompted by this blog post, which was in itself presumably prompted by this HN item)
Yesterday was a nice day, so I decided to walk from Derby to Alfreton (roughly here). The criteria were pretty simple:
Ignoring “automatic routing” altogether, what map provider in the UK has that data? OSM does (at least in the area that I’m interested in). Google sort-of does - I’m sticking to roads here, and via their StreetView I can see that e.g. Codnor Denby Lane does have somewhere to safely walk, but it’s not really practical to “virtually walk” an entire route in Google’s StreetView to check it before setting off. HERE has some aerial imagery but not with enough detail. The UK’s Ordnance Survey surely has most of the data, but getting at it is hard (even if you pay them money). The small map on this page can be zoomed in to hint at sidewalk info, but neither the buyable OS Explorer map (which I have a copy of) nor this page which I presume is based on OS OpenData, do. Bing can show basic road info and their own or OS Explorer’s map style, but no “is this walkable” information.
With regard to POI data, Google was (until yesterday) the only one that had the correct name for this pub, and does link through to food and beer details (even CAMRA, usually the go-to guide for this sort of thing, are out of date). However, there’s no differentiation on the map beyond a “place that serves food” icon and a mousover blurb (“family friendly chain pub and restaurant”).
So am I missing something - is there anyone out there with this level of data that’s available in an accessible format, other than OSM?
There’s a discussion over on talk about contributions to OSM’s Standard map style. This diary entry explains what I did in order for a change to fix a problem with bus guideways that was visible in that style. It’s provided here so that hopefully it can help more people that want to (but don’t necessarily know how to) to contribute.
The first thing I did was log an issue for the problem that needed to be fixed. As well as describing the problem that also outlined the fix (actually I referenced some code in a different map style where I’d already fixed the problem). The initial discussion on the issue was helpful in that other people said that yes, they thought that it was a problem too.
However, describing the problem is only part of the way towards actually fixing it. In order to do that, I started by reading the “contributing” guidelines and the links from there.
I already had a github account. I forked OSM Carto into my area to create this. Within there, I created a new branch “guideways_low_zoom” (since deleted) just by typing the name in into the github UI.
I already had a virtual machine running Ubuntu 16.04.3 LTS (server) with all the pre-requisites in place. I’d set it up as per the switch2osm guide, but mainly use it for developing a different map style based on an older version of OSM Carto. Another option is to follow the recommendation in the OSM Carto repository and use Docker.
I then had to “git clone” my forked repository to set it up on that machine, and I then did:
git fetch --all
git checkout --track origin/guideways_low_zoom
to get the correct branch there.
In order to actually get the vanilla copy of OSM Carto working, I had to follow the instructions in the “Shapefile download” section of the switch2osm instructions (essentially running “scripts/get-shapefiles.py”).
I’ve finally got a reasonable rendering of linear barriers working - like the cattle grid in this picture:
That’s here in OSM and you can see a zoomable map here.
The key to getting it working was to ensure that the width of lines used at high zooms mostly matched the real width of the feature on the map (see here for cattle grids), except for gates and stiles which are designed to stand out slightly. Walls and fences are slightly different, allowing you to tell them apart.
It’s been possible to run Linux software on Windows for some time using Microsoft’s “Windows Subsystem for Linux”. Here’s what needs to be done to run a simple tile server there, based on the existing instructions here .
Firstly, install “Bash on Ubuntu on Windows”. There’s are some intructions here (but don’t worry where that says “This won’t work with server software”). Continue down to where it says “You’ll be asked to create a user account and password for use in the Bash environment”. It’s easiest if that user account is the one that you use in the rest of the switch2osm instructions.
Once installed, at a shell prompt “lsb_release -a” should say:
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.3 LTS
Release: 16.04
Codename: xenial
What you get is surprisingly similar to a standard Ubuntu server machine. Some things don’t work (e.g. “screen”), and you’ll have to manually start some of the services that will need, but otherwise things are very familiar.
Run “sudo apt update” and “sudo apt upgrade” to upgrade to the latest version of everything.
Next, go over to the switch2osm instructions and follow those as normal, except for the caveats pointed out below:
If you want to ssh in (you probably will) you’ll need to “sudo service ssh start”.
You’ll likely want to allow access to ports 22 (for ssh) and 80 (for http) and possibly others at the Windows Firewall (inbound rules / new rule / port / tcp port 80 / allow / all / “name”).
After installing postgresql but before doing e.g. “createuser renderaccount” you’ll need to start postgresql manually. “sudo service postgresql status” will say “down”, but after “sudo service postgresql start” it will say “Starting PostgreSQL 9.5 database server” and “service postgresql status” will then say “9.5/main (port 5432): online”.
A previous diary entry explains how I created a basic map legend for a map that I maintain that highlights various things common near me that tend not to get shown very well elsewhere - public footpaths, hiking trails, offices, that sort of thing. It struck me that a lot more things had names than almost any other map (other than OSM Bright) actually shows.
I therefore went through a process of looking to see what tags people used names with (it’s almost everything - here’s a named cattle grid), and what they used as the “naming tag” for various things (mostly “name”, obviously). The result is here:
although to get a better idea of things click here and zoom around.
You might want to set up a rendering server for a short period of time, or you might want a server that has lots of memory while you’re importing data, but much less afterwards. An “always on” Azure server will probably cost more that a comparable VPS somewhere else, but if you only want it for an hour or two at a time, it may cost considerably less.
This page describes using Microsoft Azure to do this, but of course other cloud providers are available, and depending on what you want to do may be better suited to your needs.
I have no commercial relationship with Microsoft (beyond occasionally buying hardware with Windows preinstalled and then later removing it) and this isn’t a “recommendation”, beyond providing another option to setting up a server yourself.
If you haven’t already got one that you want to use, create a Microsoft account. Jump through whatever validation hoops you have to (for me that was an email address or a phone number, but this may vary depending on where you are).
Next, go to portal.azure.com and sign in. There will probably be a few “free trial” buttons around; click one of them to get a 30-day trial of up to £150 / $200. You’ll need to provide the Microsoft account details (from above), a phone number (again) and a credit card. If you’re in Germany and want to keep your data there see here.
This page is a good starting point. You’ll want to install the Azure CLI because it’s a lot less fiddly than than the alternatives, though there is a “Cloud Shell” option in the portal (actually that’s an account on a remote Ubuntu 16.04.2 LTS VM!) and a “clicky pointy” interface in there too.
First, create a “resource group” to tie everything associated with this machine together, and make it easier to tidy up later. I created this in “uk south”, you can pick somewhere closed to you:
az group create -n MyResourceGroup -l uksouth
There’s recently been a thread on the talk-gb mailing list where someone has decided that, despite previous custom and practice there, the “name” field in both English- and Welsh-speaking areas of Wales should be a compound of both the English and Welsh names. No-one says “I’m climbing up Snowdon / Yr Wyddfa today”, they’ll use one name or the other, not both together.
In the Welsh-speaking areas the Welsh names are more likely to be used; in the English-speaking areas the English names. It’s not a hard-and-fast rule; this peak in the Black Mountains is referred to about equally by both the Welsh and English names, despite it being in a predominantly English-speaking area.
Wikipedia gives an idea of Welsh-language take-up here. That’s a bit broad-brush; for example I don’t think there’s an isogloss between Carmarthenshire and Swansea where people gain/lose the ability to speak Welsh.
So how is it possible to extract data from OSM with the Welsh name in the Welsh-speaking areas and the English name in English-speaking ones, both when creating e.g. a rendering database for the first time and when updating it as people update OSM? Firstly we’ll just consider the “loading the database” part.
There are a couple of possible solutions to the problem. I used “osmosis”, which has a handy “tag transform” feature. The Welsh one is here; the English one is similar.
Very roughly, the Welsh-speaking area of Wales corresponds to this area. That’s not perfect, but it’s not a bad approximation for a rectangle. I downloaded the latest Welsh data from Geofabrik and cut that area out of it:
osmosis --read-pbf wales-latest.osm.pbf --bounding-box left=-4.82 bottom=52.02 right=-3.34 top=53.69 --write-pbf wales_cy_before.pbf
Convert the “Welsh-speaking” part to names based on “name:cy”:
osmosis --read-pbf wales_cy_before.pbf --tag-transform transform_cy.xml --write-pbf wales_cy_after.pbf
Create a copy of the larger file with names based on “name:en”:
I’ve updated the switch2osm “Manually building a tile server (16.04.2 LTS)” page to use the latest version of the osm-carto style, now that the issues underlying this problem have been fixed.
This means that it should be possible to follow these instructions to set up a tileserver from scratch that creates tiles in the same style as OpenStreetMap’s “standard” style in just a few hours (for small areas the longest period of the setup is waiting for the shapefiles used by the style to download).
What would be really useful would be if people could test that these notes do actually work for them - is there anything that should be made clearer, especially to someone who might not be familiar with Linux at all?
I’ve created a map legend for the map style that I use. It’s a set of map data that can be appended to a rendering database to create a map legend in a blank bit of map (actually in the middle of Australia).
There are currently two parts - linear features and POIs. As far as the map is concerned these are “real POIs” (though they’re never added to OSM itself of course - just added to the rendering database) so you need to zoom in to see some of the POIs such as the one for vending machines, which only appears in this map style from zoom level 19. You could use it with other map styles, but some of the features are dependent on this lua script.
Here’s a preview of what the “roads” part of the legend looks like:
but to use it properly follow the link and move around.
I’ve written before about the changes needed to render more zoom levels than 18 with a “Switch2Osm-style” tile server.
However, sometimes zoom level 20 isn’t enough. Here:
is part of Nottingham at zoom level 20. At least one of the office names doesn’t appear (it corresponds to here in OSM). The problem is that the way that the “standard” renderd stores metatiles means that only a certain number of tiles can be stored for each zoom level (see this list post for the details). In order to store more I changed renderd slightly so that more zoom levels can be stored - see here and here for the details.
Rendering works fine at higher zoom levels (up to 28 in my example) so that all of those office names now appear. Here’s the same area at zoom level 21:
Although plenty of maps have displayed long distance footpath information for ages, I’d always assumed that it would take a lot of extra processing to get working. Turns out I was wrong:
Because of the way that relations are handled within osm2pgsql, converting route relations into “just another highway type” is pretty straightforward, and once created that new “highway=ldpnwn” can be added to the stylesheet.
In this case the two things to think about are the fill and the text. The “fill” is a short wide purple dot separated by a long gap from the next one. In the code (see the links above) at “line-dasharray” the “1” says “short and the “60” says “separated by a long gap” and further down the “line-width: 4” says “wide”.
On many occasions in the past I’ve said “it’s really easy to create your own Garmin maps” but I don’t think I’ve ever documented the procedure. This is an attempt to do that.
You’ve got some sort of Garmin device with an SD card in it with some maps that you’ve downloaded from somewhere on it.
You want to replace those with a map you’ve created yourself.
You’ve got a PC with “enough” memory in it. The one I used for the examples below had 4Gb; it’s possible that less may be needed for a small file such as the one used in the examples below.
All the path examples below for for Windows (actually tested on Windows 7), but it should be relatively straightforward to adapt them to work on other operating systems.
The easiest way to create Garmin maps from OSM data is to use “mkgmap”. That’s available from:
The download page for the latest mkgmap is http://www.mkgmap.org.uk/download/mkgmap.html and the current version is “3676”.
Unzip the zip file and move the folders intact to a known location. In my case I moved it so that “mkgmap.jar” file was in: C:\Utils\mkgmap-r3676
I didn’t use a path with a space in it or a path that Windows recognises as “special” to avoid any possible confusion later.
“Splitter” is a separate program used to split a large OSM data file into portions small enough for mkgmap to work with (the resultant Garmin map can still route between portions).
The latest Splitter is here: http://www.mkgmap.org.uk/download/splitter.html and the current version is “437”.
Unzip the zip file and move the folders intact to a known location. In my case I moved it so that splitter.jar was in: C:\Utils\splitter-r437
Next, we need to download an OSM extract to use to create maps from. As an example I’ll use Romania, and extract for which can be downloaded from here:
http://download.geofabrik.de/europe/romania.html
That page currently says:
A while back I described how I was showing tree types in woodland. The “unfinished business” there was “what about forest areas where the trees have been cleared?”. Mapping of that is a bit hit and miss. “Forestry” has been suggested, but doesn’t have many takers, and “forest” is actually often used for “the entire forestry area” (at least where I’m interested in rendering tiles for - I suspect it varies considerably worldwide). The wiki page and the standard style rendering discussion don’t distinguish, but I thought it was worth trying to separate out “natural=wood” and “landuse=forest” where the latter is used for “the entire forestry area, including where there are currently no trees”.
Here’s the result:
