… See full entry

… See full entry

Added detail from my Mapillary photos and imagery

Did you know that OSM data is available as an open dataset on Amazon Web Services? Updated weekly, the files are transcoded into the .orc format which can be easily queried by Amazon Athena (PrestoDB). These files live on S3 and anyone can create a database table that reads from these files, meaning no need to download or parse any OSM data, that part is done!

In this post, I will walk through a few example queries of the OSM changeset history using Amazon Athena.

For a more complete overview of the capabilities of Athena + OSM, see this blog post by Seth Fitzsimmons. Here I will only cover querying the changeset data.

1. Create The Changeset Table

From the AWS Athena console, ensure you are in the N. Virginia Region. Then, submit the following query to build the changesets table:

CREATE EXTERNAL TABLE changesets (
    id BIGINT,
    tags MAP<STRING,STRING>,
    created_at TIMESTAMP,
    open BOOLEAN,
    closed_at TIMESTAMP,
    comments_count BIGINT,
    min_lat DECIMAL(9,7),
    max_lat DECIMAL(9,7),
    min_lon DECIMAL(10,7),
    max_lon DECIMAL(10,7),
    num_changes BIGINT,
    uid BIGINT,
    user STRING
)
STORED AS ORCFILE
LOCATION 's3://osm-pds/changesets/';

This query creates the changeset table, reading data from the public dataset stored on S3.

2. Example Query

To get started, let’s explore a few annually aggregated editing statistics. You can copy and paste this query directly into the Athena console:

SELECT YEAR(created_at) as year, 
      COUNT(id)        AS changesets,
      SUM(num_changes) AS total_edits,
      COUNT(DISTINCT(uid)) AS total_mappers
FROM changesets 
WHERE created_at > date '2015-01-01'
GROUP BY YEAR(created_at)
ORDER BY YEAR(created_at) DESC

I will break down this query line-by-line:

… See full entry

City of Yathrib

This is a quick count of the various editors used by mappers in 2020. iD is on top with 7.6M edits followed by JOSM with 5.6M. The data was collected by looking at tags->created_by in the changeset database.

Editor	Count
iD	7636890
JOSM	5644149
StreetComplete	414353
MAPS.ME	160365
Go Map!!	122639
OsmAnd	118465
Potlatch	115427
Vespucci	100327
Other	90879
osmapi	73587
Refill Südtirol	22612
Services_OpenStreetMap	16985
Merkaartor	13137
autoAWS	11626
All the Things	10548
caresteouvert.fr	7930
Go Kaart!!	7567
FireYak	6956
OsmHydrant	6564
MapRoulette	3206
ArcGIS Editor for OpenStreetMap	2611
Rosemary	2067
Gnome Maps	1437

The URL is http://taginfo.openstreetmap.pe.kr/. I want to set my site’s domain taginfo.openstreetmap.kr, but can’t because someone already has openstreetmap.kr domain.

… See full entry

This will be my third and final installment in a series of articles discussing device I’ve used when preforming surveys when contributing to OSM.

In the first article, I talked about accuracy limitations associated with GPS signals while the second gave an overview of four specific mapping devices I’ve personally used. In this article I’ll compare the accuracy of three devices previously discussed using GPS traces.

Test Setup

To test device precision, I recorded GPS traces along a hike with a variety of line-of-sight obstructions to the sky with each device recording at the same time. The traces were assigned colors to help differentiate each device:

• Samsung Galaxy S10e - Orange
• EcoDroidGPS - Green
• Garmin GPSMAP 66sr - Purple

The arrangement of the devices are as follows:

• The EcoDroidGPS was placed in a backpack with the reciever on an antenna mast above my head.
• The Garmin was attached to one of the backpack straps with the built-in antenna stub poking just above my shoulder.
• The S10e was in one of the side pockets of the backpack.

The EcoDroidGPS and S10e location information were sampled every one second within a GPX track using the app OsmAnd. The 66sr records as an “activity” with location information sampled every one second. The resulting GPX track was downloaded from the device.

Results

The Bridge

Before arriving to the trailhead, I had to walk along the shoulder of a bridge spanning a medium size river. The bridge is long enough to offer a best case scenario for accuracy. There is a good 45 degrees of sky visible from most angles in the center of this bridge.

The result of a stretch of road with minimal interference is quite predictable, all three traces are stacked right on top of each other without significant deviation.

… See full entry

Found that islands around Gruney in Scotland not yet mapped. It is a little bit strange.

After being inactive for a long period, I am back to contributing again to OpenStreetMap. The once-poor Internet infrastructure in my area has hampered my online mapping in the last five (5) years. Fortunately, the connectivity has improved a lot now (albeit intermittent at times, as one would expect in a mountainous terrain that generates its own, unique weather system affecting signal quality). Another factor that discouraged rigorous mapping here in the North is the long absence of updated satellite imagery. Thank mapping gods, we now have updated satellite data. Yey!

I haven’t done much, except some minor islands and adjusting coastlines. Hope to add more.

In this article, I’ll continue sharing my experience on finding a generally-available commercial device to be used with preforming surveys for contribution to OSM. I previously wrote about GPS precision and why, by itself, it is limited to meter precision.

Below are the devices I’ll be discussing based on my own usage. I’ll share how I used them and my thoughts on each. In a final article of this series, I’ll provide a GPS trace comparison of the EcoDroidGPS, smartphone, and Garmin GPSMAP 66sr as well as which device I’d pick if doing it all over again.

• Modern smartphone (Samsung S10e)
• EcoDroidGPS
• Garmin GPSMAP 66SR
• DIY (simpleRTK2B or Arduino solutions)

Modern smartphone (Samsung S10e)

The first device I used for surveys and general contributions to OSM was with a smartphone. I currently use a Samsung S10e with the below Android apps depending on the complexity of my contribution:

• OsmAnd - Used for adding basic POIs and creating GPS tracks with built-in plugins.
• StreetComplete - Used for casual contributions and tag refinement on existing features.
• Vespucci - Used for more complex editing of most features in OSM. Offers an editing experience like iD or JOSM, except on mobile.

Not surprisingly, since I generally have my phone with me, this is the most convenient when I stumble across mapping opportunities in the wild. The location accuracy is acceptable. It can be used to map larger roads and general POIs without issue, but is not accurate enough to trace a sidewalk or trail. I’ve also noticed that accuracy improves in urban areas likely due to the ability for Google’s location services to supplement GPS data with nearby WiFi access points and Bluetooth.

… See full entry

A common task with OpenStreetMap data in PostGIS is to convert polygons to points to place labels. For simple polygons, the centroid can be used, but some shapes like C-shaped polygons, the centroid can lie outside the polygon, so ST_PointOnSurface is used. This function guarantees the point returned is within the polygon.

The only issue with ST_PointOnSurface is that it throws an exception on some invalid geometries. This isn’t a problem with a database created by a recent version of osm2pgsql which only creates valid geometries, but for older versions or other data loaders it’s unacceptable. This has lead people to writing wrapper functions that check the validity or catch the exceptions, but I’ve seen no benchmarking of the various options.

To benchmark the options, I loaded the planet data from 2020-10-12 and looked at named water polygons - those that matched ("natural" = 'water' OR waterway = 'riverbank') AND name IS NOT NULL. To make the system better reflect a tile server under load, I set max_parallel_workers_per_gather to 0 and jit to off. I then ran the query EXPLAIN ANALYZE SELECT function(way) FROM planet_osm_polygon WHERE ("natural" = 'water' OR waterway = 'riverbank') AND name IS NOT NULL;.

I tested with ST_Centroid, ST_PointOnSurface, ToPoint from postgis-vt-util, a function that checked validity before calling ST_PointOnSurface, a function that caught the exception from invalid geometries, and a function that used ST_Centroid for polygons with 4 corners and ST_PointOnSurface otherwise. The definitions are at the end of this post.

Function	Time
ST_Centroid	277s
ST_PointOnSurface	408s
ToPoint	575s
point1	568s
point2	409s
point3	409s

Parallelism

… See full entry

I’ve spent a fair amount of time looking for the “perfect” consumer device to be used in my on-the-ground surveys that provides better-than-average accuracy without spending thousands on survey-grade equipment. This article series is an attempt to catalogue my experience to those who are interested.

Before diving in, I’d like to review the obstacles to accuracy when surveying using GPS-enabled devices. While this subject may be common knowledge to those familiar, it took me a considerable amount of time to understand some of these points and was a major driving factor in my quest for the “perfect” device.

GPS vs. GNSS

Throughout this article I’ll use the term “GPS” to refer to the systems and satellites that provide longitude and latitude location information to a user. Please note that this is not 100% accurate, however, since “GPS” is a specific satellite (known as a “constellation”) location system among many. In fact, GPS is part of a general Global Navigation Satellite System or GNSS. Other GNSS systems are as follows:

• GPS
• GLONASS
• Galileo
• BeiDou

I’ll be using “GPS” in an attempt to minimize confusion to those (like myself) who were unfamiliar with this distinction until now.

Precision Limitations

In brief, GPS works by analyzing a time code that is continuously sent by satellites in orbit around Earth. Receivers take that time code from multiple satellites and determine how long it took for the receiver to receive the time code data. This information is then used to calculate a longitude and latitude location on Earth.

… See full entry

OSM RELATED ACCOUNTS

OSM help OSM Forum OSM Wiki mapillary OSC How I contribute to open street map

Between 7 October and November 2020, a series of tropical cyclones and seasonal monsoons caused heavy rainfall, floods and landslides in Central Vietnam. About 230 deaths and 53 missing were reported in the national media.

With the help of Russell at HOTOSM, I created this project to contribute map data of the districts where the highest number of deaths occurred. These districts mainly lie in the mountainous part central region and therefore more prone to natural disasters. Although open to Beginner mappers, this is more accurately an intermediate difficulty mapping experience.

URL: https://tasks.hotosm.org/projects/9734

Help map flooded areas in Vietnam! Cyclone Molave brought heavy rains, widespread flooding and at least one major landslide to Vietnam.

BushmanK?

I love reading his rather insightful diary entries regarding OSM. Rub some people the wrong way no doubt, although I agree a lot with what he says, especially things about database stuff.

Check out the OSM apps catalog. It brings together and shows all the great work (mapping and programming) that we have done.

Help improve it by updating the software documentations (screenshot and description) in the osm wiki.

Exploring the growth and temporal mapping patterns in OSM in North America

The following figures are from my OSMUS Connect 2020 Talk. Additionally, I’ve included the relevant queries to reproduce these datasets from the OSM public dataset on AWS (See this blog post). For this work, I used a bounding box that encompasses North America.

Starting with the big picture…

This year we are averaging about 900 active mappers each day, with significant growth in the past few years:

SELECT 
    DATE_TRUNC('day',created_at) as day,
    COUNT(DISTINCT(uid)) as user_count,
FROM changesets
WHERE min_lat >  13.0 AND max_lat <  80.0 AND min_lon > -169.1 AND max_lon < -52.2
GROUP BY DATE_TRUNC('day',created_at)

How did we get here?

This next graph quantifies a mapper’s first edit in North America by month. For example, in August 2009, 1,700 contributors edited in North America for the first time. In January 2017, close to 7,000 contributors edited in North America for the first time.

… See full entry

1. Know your source data.
2. Map only what you know.
3. Your goal is improvement, not completion.
4. Fewer, more accurate nodes are better than many less accurate nodes.
5. All lines are straight, all curves are circular, and all corners are perpendicular until proven otherwise.
6. Map one thing and map it well.
7. Don’t be afraid to delete inaccurate nodes imported from TIGER. The import from 2007 won’t care.
8. Make your editor work for you.
9. Satellite imagery can hide mistakes and missing features. Take a step back and look at the rendered map.
10. Validate and sync often.
11. Nothing is ever named “State Route”
12. Structural and semantic accuracy are more important than visual accuracy.