I spent most of my time today adding sidewalks in the area of Waterloo where my parents live, basically in an area boxed by Brookridge Drive, Kimball Avenue, Ridgeway Avenue, and 9th Street. There are some other areas of Waterloo that have sidewalks and crossings mapped, but they tend to be near downtown which, while useful, is a small percentage of the sidewalks that exist in Waterloo.

I actually haven’t been in the area since Easter weekend, but between the satellite imagery and my own memory I think I did a reasonable job mapping the sidewalks and crossings. I actually didn’t know about putting a node where a sidewalk/crossing and the road it’s crossing intersect until I got well into it, but I’m glad I went back and read the OSM Wike entry on Key:crossing as I hadn’t been creating the intersecting nodes until I did. I haven’t seen the results in OsmAnd yet (which is where I’ve seen crossings here in Champaign), but I’m pretty sure they’ll show up now that I’ve added them.

Admittedly part of the reason why I’m mapping these sidewalks is to direct some attention to where there aren’t sidewalks in Waterloo, but in my opinion there should be. A good example is here where the sidewalk on the north side of Park Lane ends after the Kimball Avenue access road crossing, but then starts again by Brockway Road, the next street to the west, only to end again at Colby Road. Gaps like this exist all over southern Waterloo, forcing pedestrians to either cross twice to stay on the sidewalk, move onto the street, or cut through the grass where the gap is. I’m hoping that by mapping the sidewalks I can highlight these gaps and maybe motivate people still living there to get them filled in.

Received some valuable feedback from the imports mailing list on the matters of data quality and the expectations on someone’s level of OSM experience before executing large scale automated data imports. I was pretty much well set in terms of data quality concerns, but it looks like I would need a bit more hand-holding from more experienced mappers and importers to properly execute a big import.

This is not a problem, though. It’s very reasonable and thankfully not a deal breaker to me because I chose a scope small enough that it’s feasible to execute this import manually instead of automated. In fact, my previous analysis that the VSI had about 570 coffee/café related business was an overestimation because - rookie mistake - I forgot to deduplicate by survey period.

The new numbers are:

• OSM Nodes Matching Coffee/Cafe: 574
• VSI Stores Matching Coffee/Cafe: 278
• OSM Nodes within 10 m of a VSI Store: 28
• OSM Nodes within 25 m of a VSI Store: 195

So yeah, lots of nearby matches to investigate. Now is the time to start fuzzy matching the business names and SK53 provided me some good reading material on that. It will be a bit challenging to do that with pure SQL (I’m trying to use dbt + BigQuery only for now), but I think it’s worth a try.

Foreword

I live in the St. Ann’s electoral ward of the City of Nottingham. As well as wishing to be able to improve the coverage of old_name + start_date for each street in Nottingham, I am intrigued to be able to discover when streets were laid out, metalled, drained & provided with sewers. Today that all seems normal, but I was astonished to discover that Blue Bell Hill Road had no street drainage nor sewers until the 1970s; a friend in Dowson Street has a well in their basement, whilst their road also has zero street drainage nor sewer, plus no water main through the street (water supply, sewer + drainage only at the rear of the terrace).

Like many cities in the UK, Nottingham has suffered shed-loads of physical upheaval/churn across the years. That has led to the appearance, alteration, disappearance and/or reappearance of streets and thus of street-names. I’ve recently gotten access to definitive information on (at least some of) those changes, and decided that I should strike the iron whilst it is hot. This diary is going to concentrate on local streets + national communication (rivers/canals/railways) as they apply to Nottingham City; it will also filter in items of national importance that occurred in Nottingham and/or affect the whole UK.

First, here is the OpenStreetMap Wiki on names.
Second, the principal sources I’ve been using:

… See full entry

I was finally able to start reconciling the Vancouver Storefronts Inventory (VSI from now on) and the OSM nodes. VSI has 578 coffe/café matches, OSM has 574. These numbers are so close, it gives me hope.

When searching from nodes in OSM that have a nearby (<10 m) node in VSI, 54 results come out. Of those, 51 are perfect matches (business name in OSM is the same as in VSI, except for things like “Starbucks” in OSM vs “Starbucks Coffee” in VSI). This isn’t too thrilling, but honestly a near 10% perfect match from the get go is pretty sweet.

Using 10 meters is pretty bold, so I’ll experiment a bit on a healthy threshold that gives me more matches but doesn’t yield too many false matches. A 25 m radius already jumps to 391 matches and a 50 m radius gives 705 which is obviously too much.

If I have the time, I should also probably start getting fancy with fuzzy matching business names to get the obvious non-identical matches out of the way so I can investigate proper mismatches.

This article is also available in Taiwanese Mandarin (台灣華語) and Taiwanese Hokkien / Taigi (台文)

---

The OpenStreetMap Taiwan Community (OSMTW) is pleased to receive the Wikimedia Alliance Fund for procuring two Insta360 One X2 (and accessories), as well as holding at least six Expeditions and Post-expedition Mapping Workshops from March 2022 to February 2023. OSMTW members will initiate surveys to the street-view terra incognita with a 360-degree-camera-mounted vehicle, then edit on OpenStreetMap and upload media taken throughout the exploration to Wikimedia Commons.

The path of this expedition differs from last time and headed south for Yingge, Sanxia, and Daxi rather than the Northern Coast (Keelung, Jingshan and Wanli). The “Street view car” dispatched this time also departs differently. (The vehicle author droves kick off at Hotai Easyrent Xindian, so he took a photo of the unphotographed Exit 2 of MRT Xindian District Office Station and uploaded to Wikimedia Commons before departure.)

… See full entry

… See full entry

Getting familiar with the Vancouver Storefront Inventory dataset. Apparently there are around 578 businesses with names that include Coffee or Cafe/Café, which looks pretty good. Judging by name only is pretty unreliable, but at least the vast majority of the matches are in the Food & Beverages category which is reassuring.

By chance, one of these businesses is already permanently closed (according to other sources): Café Logos. It was literally the first one I randomly selected to investigate, and it’s already data that should not be imported to OSM. Oh well. This is going to be tough.

On the OSM front, today I learned how to use the BigQuery dataset that has a handy table containing the OSM areas/ways as GDAL objects, which I can use to further sub select the nodes that are inside the region of interest (Vancouver) using ST_DWithin.

My last fortnight has been spent updating the photo-URLs within these diaries of pictures that I’ve taken whilst mapping. By default the Mapillary page will normally show a small version of any photo in it’s GIS-database, whilst behind it is a map using OSM-mapping; here is an example of that, showing the front of a business called AST, as it was displayed in a 12 July 2019 diary.

In the past the owner of the photo — the person that uploaded it to Mapillary — was given a button that would give them a URL that allowed them to download the raw file of the original photograph. That was useful for webpages such as these diaries, since it allowed the photograph to be displayed. However, Mapillary has changed it’s policy on the usage of those download URLs.

At some time in the past Mapillary changed the URL format for both map-display pages & download-file pages. The original map-display URLs had the following format (1st line below) whilst the download-files were 2nd-line below (the IDs in each URL were identical for a specific photo):

• https://www.mapillary.com/map/iM/<12-digit-alpha-ID>
• https://d1cuyjsrcm0gby.cloudfront.net/<12-digit-ID>/thumb-2048.jpg

Mapillary changed the format to the following (sorry about this):

• https://www.mapillary.com/app/?pKey=<16-digit-numeric-ID
• https://scontent-man2-1.xx.fbcdn.net/m1/v/t6/<150-digit-alpha-ID>?stp=s2048x1152&ccb=10-5&oh=<61-digit-alpha-ID>&_nc_sid=122ab1

Mapillary also made 3 crucial extra decisions:

1. The map URL would auto-rewrite via a 302 between the old & new format
2. The old Download URL would NOT rewrite to the new
3. The new Download URL would timeout after 14 days (or maybe less) (appears to be just 2 days)

Thus, after 14 days of trawling through every relevant page & changing all relevant URLs the photos are still not showing. I’m not happy.

And here the relevant photo to see what happens:

… See full entry

I tried to launch JOSM for the first time in a little while & got the error:

failed to execute josm-latest no such file or directory

It had worked fine the last time I used it & had been continuously updated every since.

Searching for the Fix

An internet search did not reveal anyone reporting the same error, but did point me towards the GitHub site (more on that later, with the eventual fix at bottom of this diary post). Searching the computer did not help much, but it did reveal the .desktop menu file:

$ locate josm-latest
/etc/default/josm-latest
/usr/bin/josm-latest
/usr/share/applications/org.openstreetmap.josm-latest.desktop
/usr/share/josm-latest/josm-latest.jar

My local .desktop menu file is identical to the GitHub-code latest .desktop file. I tried running the exec-line in that file from a console (I work under Devuan, which is a Linux distribution):

$ josm-latest %U
bash: /usr/bin/josm-latest: /usr/bin/bash: bad interpreter: No such file or directory

Here is the reason why:

$ file /etc/default/josm-latest
/etc/default/josm-latest: ASCII text
$ cat /etc/default/josm-latest
# Options to pass to java when starting JOSM.
# Uncomment the JAVA_OPTS lines to enable their use by /usr/bin/josm-latest

# Increase usable memory
#JAVA_OPTS="${JAVA_OPTS} -Xmx2048m"

# Enable OpenGL pipeline (2D graphic accelerators)
#JAVA_OPTS="${JAVA_OPTS} -Dsun.java2d.opengl=True"
$ file /usr/bin/josm-latest
/usr/bin/josm-latest: Bourne-Again shell script, ASCII text executable
$ head -1 /usr/bin/josm-latest
#!/usr/bin/bash
$ la /usr/bin/bash
ls: cannot access '/usr/bin/bash': No such file or directory
$ which bash
/bin/bash

(translation): Neither josm-latest provided within the latest JOSM is fit for purpose:

… See full entry

Wikidata is a free knowledge base for linked open data designed to support Wikipedia and its sister projects, such as Wikivoyage. It contains over 97 million entries structured as a “Labeled Property Graph,” which is more powerful than RDF-based graphs. Like OpenStreetMap (OSM), Wikidata (WD) is an open crowdsourcing project with a large and active community.

Since 2014, OSM can be linked to WD through its tags. Currently, there are about 5.5 million such Wikidata tags with steadily growing popularity. These links can be used to create interesting products, for example a map with castles enriched with factual data from WD. However, the quality of these manually captured links in OSM is as yet unknown and untested. One must also note that the preferred way from WD to OSM - the other way around - is to use only coordinates (WD property P625) - i.e., no WD properties such as P402 are to be used because this covers only OSM relationships.

Now, two computer science students, Jari Elmer and Timon Erhart, from the University of Applied Sciences of Eastern Switzerland (OST), with the help of Sascha Brawer - a young software engineer in “un-retirement” and Wikipedian - have developed an application called “osm wikidata quality checker”. The goal was to check the existing links from OSM to WD. The errors found - for example invalid WD entries in OSM - are also sent to osmose with a suggested correction. Osmose is a quality assurance tool for detecting problems in OSM data. The goal of the application was to become an integral part of OSM’s quality assurance ecosystem. It handles the large amounts of data in the two databases (about 1.5 TB each).

… See full entry

Date - 07/06/22
Time Spent - 0.75 hrs Activity:
1. Trying to decide focus of next steps. Pedestrian footpaths or Lane mapping accuracy?
2. Tried to evaluate if small improvements to pedestrian footpaths can make it easier for kids/parents to walk to school?
Notes:
Used geojson.io to create a map_potentialwalkway.geojson trace to see where a footpath trace can be drawn. More to come about this exercise.

Next week we’re having an OSMLondon pub meet-up for the first time in a while. Or at least I am. People don’t seem to like setting themselves as “attending” on osmcal.org, but I don’t think I’ll be alone in the pub. Looking forward to it anyway!

Today I wanted to print out a map of “Parkland Walk”, a local nature trail (and former railway). This was a project for/with my 6 year old son, which I spent a bit longer on than I should have today. In his class they’re doing various activities related to Parkland Walk. I thought it would be fun to give him a big map in style which he could colour in.

… See full entry

I’m preparing a video tutorial on thatched buildings - it’s dead simple, but it’s an interesting topic, I thought. So I was doing a bit of research trying to learn something about local thatching traditions and came across a 1994 survey by - it turns out - a thatcher. I only came across one volume which is basically a photo album of 106 thatched buildings in Co. Kilkenny with handwritten captions given the location. Location being in most cases the townland. But because it is handwritten and because this is Ireland, I have not been able to identify all the townlands.

What I’m doing now is trying to find the townland and trying to identify the building. Most times, if there are other buildings in the picture, it is possible to spot them in the townland by comparing the arrangement of buildings. Luckily, most buildings in Co. Kilkenny are mapped thanks to our #osmIRL_buildings project. And luckily, those are mostly old buildings, so I don’t have to worry about them being built since 2019. The National Index of Architectural Heritage is somewhat helpful in that they have indexed some of those buildings, but not all. They have a map where you can find a blue dot for those marking the spot. However, they are not always correct. They also have pictures of those buildings (sometimes also the wrong ones) which I can compare to that survey/ photo album. Sometimes, rarely, because those are very rural areas, I have mapillary to work with.

If they are not too far away and accessible for a non-driver like me, I’ll go and check them out in situ, take mapillary and a photo or two for Wikimedia. #OpenData, baby!

You can kind of see on Esri World Imagery (Clarity) Beta, whether it is thatched or a slate roof, because the thatched roofs have smoother corners and the colour is a bit different.

… See full entry

I have an endgame in mind, which is having a complete walkability study on a bunch of major cities in the world, which competes in quality with walkscore.com but it’s fully open data.

That’s too grand for me to accomplish in a year of less-than-part-time effort. So I’ve decided to scope it down to a single city I care about, which is Vancouver. How hard can it be to analyze walkability in a single city? Well, pretty damn hard actually.

The very first thing I’d like to consider analyzing walkability is proximity to amenities like cafés, markets and pharmacies. Turns out OSM seems to have pretty good coverage on shops in Vancouver already, but for me to be very confident on my analysis the study stars with an evaluation of data coverage.

If I’m going to compare OSM with an official source, say Vancouver’s Storefront Inventory, whatever the coverage might be… I might as well import what’s missing? I think I owe OSM this much, and it will be nice to say that the whole data used in the walkability study is from OSM instead of from multiple sources.

The thing is, my past experience with data imports is limited to a single one I made for Wikidata of Higher Education Institutions in Brazil, and it took a whole month to finish it. I had more tooling available (OpenRefine is very integrated with Wikidata), more time available (5 years ago I had more energy) and the data was much more straightforward (no mapping involved, just categorization)…

Considering all that, I’d estimate it would take me about 3 months to import the whole thing following proper procedures. So I’ve decided to scope it down once again, to a subset of the data that I can reasonably scan through manually. I’ll start with coffee shops / cafés only. That I think should bring the estimate back down to a single month or so. Hopefully.

Date - 07/05/2022
Time Spent - 1.5hrs
Changeset - #123254900
Activities:
1. Focused on improving junction and road definition at Bethel Road & Pickforde Dr.
2. Split Bethel Road and Pickforde Dr. by adding nodes.
3. Added lane information for Bethel Road and Pickforde Dr. going into and out of intersection.
4. Reviewed turn assignments at junction. They look okay.
Notes:
It was difficult to understand the correct approach to mapping a junction. The OSM Wiki documentation can benefit from additional examples and best practices.

Dear Diary.

I made my first edits today.

• mcvillain

In my last diary entry I wrote about putting the requested note ids into url. I said that it’s likely not going to be a problem because ids are stored in the url part that’s not sent over HTTP therefore it doesn’t matter how long is this part, how many ids it contains. That was said about a typical output of neis-one.org note feeds for countries that contain updates dating back up to a week. For a busy country that’s usually a few hundred notes. Concatenated with single-character separators they result in a few thousand character string appended to a url.

Feeds are not the only way to get note ids from neis-one.org. In fact you may no even want to get notes from the feeds because of their one week age limit. There are regular html pages for each country too, one for all notes, another for open notes. They likely contain more notes than the feeds do. The open notes pages, which are of interest to someone who wants to resolve notes in a given country, can have up to ten thousand notes. I added options to open those with note-viewer, along with arbitrary html files. The last two options in the neis-one.org Get … notes for this country dropdown of the XML tab set up the selector (cells in second columns of tables from neis-one.org webpages) and open the webpage that you have to save to a file and then open that file with note-viewer. Unlike the xml feeds, even Firefox won’t start the download automatically.

… See full entry

“OSM Latest changes” (https://rene78.github.io/latest-changes/) got a few improvements:

Tag comparison table

… See full entry

Notes of countries

One of possible uses of note-viewer is browsing notes in some area. You can do this using the BBox dialog that lets you get notes in a rectangular area. This is similar to how the note layer on the OSM website work and how the editors like JOSM load notes. And it similarly won’t work if the area is too large or not rectangular. So if you want to get notes in a country, you won’t be able to get them easily. But you may know that there are services that show notes for a given country, like the one by Pascal Neis. There you can select a country and get a list of notes filtered by their status.

That’s how you can get a list of notes for a country. But what if you want to see notes on a map? The list doesn’t contain note coordinates so it can’t be done directly. The situation is similar to looking at a list of user’s notes on the OSM website, something that note-viewer was originally written to take care of. Can it handle country lists now? Actually it can handle any list with note ids. In addition to showing webpages with note lists, Pascal Neis’ site also has Atom feeds. It’s going to be easier for us to get note ids from these feeds. And no, unfortunately, they still don’t contain coordinates.

Any list in our case is an XML file, and note-viewer handling it means that it’s possible to come up with selectors for XML elements that contain note ids. If ids are inside some attributes, the names of the attributes also need to be specified. Click on the XML tab in note-viewer and look below the instructions for neis-one.org to see the inputs for selectors and attributes. You won’t have to use them if you only want to get note lists from neis-one.org, just follow the instructions above.

Getting the list

… See full entry

Records for phase 1

This diary is for recording my progress on the program and show the effect after every edit.

1: Z-index is added to the way with the selector [partOfActiveRoute] to make the target route float on the top of the map.

… See full entry