Recently I found Streamlit which is a pretty cool Python library that makes it easy to create web apps for visualising data.

I converted changeset dump from planet.osm.org to Parquet file format and uploaded it to AWS S3 storage. Then created this streamlit app in their free cloud: https://ttomasz-tt-osm-changeset-analyzer-main-apdkpy.streamlit.app/ which displays some basic statistics.

The app leverages the power of DuckDB, a database engine that can query these files over internet on demand. Parquet files, which are a popular format in modern cloud data lakes, have several advantages over traditional file formats. They are column-oriented, compressed, and support range requests, which means that you can download only the portion of the file you need, instead of having to go through the entire file, making processing larger datasets much faster.

DuckDB works similarly to SQLite in that it doesn’t have a dedicated server. You run the queries locally [0]. This makes the setup super simple you either install the binary or configure connection in IDE like DBeaver and you can run SQL queries.

Running these simple SQL queries over remote Parquet files takes about a minute or two. Trying to do the same with a custom script on raw changesets.xml.bz2 file would run longer not to mention that the effort to prepare the code would be much much larger.

It would be great if OSM hosted more “consumption ready” data instead of relying on users to do their own coding and parsing.

Let me know if you have some ideas for charts/tables that could be added to the demo.

[0] - well in this case they are running on streamlit cloud’s server but you can run the queries locally on the same parquet files easily

For this area beside the Blue Ridge Wilderness, I started out by adding what I knew of the Dangerous Park Trail and the Pueblo Park Interpretive Trail. There were a few miles of Dangerous Park already on the map, but they didn’t get all the way to the park. Unfortunately, the trail was diverging from what the Forest Service claims at the point I left it, so the little bit to the northern terminus includes guesswork. There’s some trail visible there.

I then worked on stuff in the wilderness and primitive area near the state line. It looks like someone has added in the trails from FS information (including attribution) in this area. I had a couple of adjustments based on my GPS, but the trail routes look good. I’m not sure if these are downloaded tracks or copied from the FSTopo. I’m seeing some changes between the two. The tracks that can be downloaded are more recent. I added signs to the mix. Guideposts and an information board. And parking.

I wanted to add the trails that connect to Dangerous Park, so I took the time to figure out downloading FS trail data again. There’s only about 4 different ways. Do they all connect to the same database or is it possible does one have to choose the right one to get the most recent data? All kinds of regulations are encoded into the tags on these trails. There’s also an indication of the state of the trail in “trail class”. Class 1 and 2 are generally represented here. Class 1 is minimally maintained and tread is intermittent and indistinct. Class 2 expects tread to be continuous, but still rough. Class 3 is continuous and obvious tread. These are trail_visibility statements! Always good to have that included.

So I got those trails added and while I was at it, I adjusted a few roads onto their route and added names and numbers. Lots of roads were called Saddle Mountain that are actually something else including a main one that is the Frisco Divide Road.

… See full entry

MapComplete has - for some thematic maps - the ability to leave a review on an entity with Mangrove.Reviews. Up till now, I had no idea how much this feature was used. However, due to technical reasons I had another look to the reviews module and discovered the ‘download all’-option on mangrove.reviews

Mangrove Clients

The analysis was made with data from 20 january 2023, downloaded around 17:00 UTC time.

This data contained 660 reviews. As the website making the review is recorded, we can make a breakdown of the top websites:

• https://mangrove.reviews is unsuprisingly the most popular website to make reviews on, with 318 reviews made
• MapComplete is the second (and the biggest ‘external’ website), with 192 reviews (of which 13 are made with the development version)
• toggenburg.swiss is third, with 35 reviews

A variety of smaller websites follows, each with a few reviews made. At first glance, most of them seem to be swiss or german. Furthermore, there are 5 reviews made by localhost:1234 and 7 by localhost:5000. The former is probably me, testing the creation of reviews while developing.

The full table is listed below.

… See full entry

In case you wanted to use my note viewer with another openstreetmap-website-based project - now you can. Although most likely you didn’t because there aren’t many of them and they don’t use notes actively. Additional projects that note-viewer is already configured to work with are:

• OpenHistoricalMap (note-viewer link) - without map tiles unfortunately, because they don’t have raster tiles
• OpenGeofiction (note-viewer link)

It’s not a surprise that notes aren’t heavily used in these projects. Often you place a note where the map diverges from reality and you can’t edit the map at the moment. But neither OpenHistoricalMap nor OpenGeofiction represent something that is currently real. Right now OpenGeofiction has less than 100 notes, and it’s not obvious if that project even needs notes. Maybe they might use notes to coordinate editing of their collaborative territories?

Now you can also edit the loaded and selected notes by commenting, closing or reopening them. This might be useful to deal with several notes at once. Actually it was one of the original plans for note-viewer. The situations where it’s helpful include someone modifying a lot of notes without a good reason. For example, users sometimes close existing notes without making any map modifications or providing reasons why the notes are irrelevant. They may do this because open notes look wrong (red with x marks) and closed notes look right (green with ticks). Closing a note may look like confirming it. Probably that’s why users sometimes close even their own notes without making any edits. Now it’s possible to quickly deal with such note modifications by searching for a given username, followed by filtering for user’s close actions, selecting all filtered notes and reopening them.

… See full entry

I’ve started a new project working with watmildon. While we were working together on applying the USGS Sq___ name changes to OSM we noticed was that there were often features in OSM that were out of sync with official name changes that happened years ago.

That got us thinking about walking through the USGS GNIS data set to find places where names had changed and OSM could be updated. After all, there are many features in OSM that have gnis:feature_id (and similar) tags that can be directly matched back to the GNIS data set.

After kicking the idea around for a while, we recently started writing some code. I’ve been working on a matching engine in C# that matches records from GNIS to OSM by Feature ID. The code also looks for likely matches where the feature name, primary tags, and geometry are close to the information from GNIS. So far the results are pretty good, but we’re still working on improving the matching.

Meanwhile, watmildon did some large scale statistical analysis on a local PBF file to look at the scale and scope of the problem. The results were very interesting!

Of the 2.3 million features in GNIS, there are only 1 million corresponding features with GNIS IDs in OSM. Some portion of these are surely existing features that just don’t have the gnis:feature_id (or similar) tags. But given our manual review of results from the matching code, there are a lot of GNIS features that are not present in OSM at all.

That’s not too much of a surprise. Some of the most common types of missing features are Streams, Valleys, Lakes, Springs, and Ridges – all things that not widely mapped in the US.

… See full entry

its what i do

As usual, I can’t quite remember how it started, but this week, I was trying to find a list of all the National Monuments of Ireland. The National Monuments Service publishes lists by county which only contain the ones under state care. These lists have made their way to Wikipedia which is a great start. They have their numbers recorded there which I have transferred to OSM under ref:IE:nm for the counties of Kilkenny, Laois and Offaly so far. (the key is recorded on the wiki, of course.)

I have sent an email to the National Monuments Service asking for a list, but so far, no reply. I probably have a reputation there by now…

However, I noticed while visiting and taking some pictures of Freshford Church that it has a black plaque (the type of which I had seen before, but never read in much detail) which stated that it was a National Monument.

… See full entry

I decided to continue trying to use JOSM for this area. I added details around the Narraguinnep Fort Historical Site, which was not simple. To add a point, I sit there in add mode and only click once so it doesn’t become a line? Hopefully that is so because that’s what I did. Then tracking down appropriate tags ended up meaning doing the same thing in iD, so not exactly a good use of time.

I continued on to details of the road around the Benchmark lookout. The track type changes halfway along. It’s nearly the boundary of the USGS map quads, so easy to miss, but they actually marked it. The road stops being improved dirt and becomes high clearance right in the middle. I did manage to figure out from JOSM how to mark that. In fact, now things are getting marked with tracktype. Smoothness was always presented, but maybe not as clear.

I decided to continue on with roads. The Forest Service marks various around the area as primary (trapezoid with an extra line markers on the map, maintained to passenger car standards) and secondary (horizontal numbers in a rectangle, should be to passenger car standards) and as 4x4 (vertical numbers in a rectangle, get the truck or even ATV). So how should one apply them? And why are they all marked as county roads, sometimes with segments with alternating numbers? None of it makes sense. I added some and lengthened some and adjusted some as I could see so their routes are all matching reality a bit better.

… See full entry

This is really just down the road from the last bit. I decided to finally try out JOSM for editing. Everyone’s doing it? It was initially harder to do the simple things. Frustratingly, it wouldn’t let me start a new line rather than adding to an old one as I started adding the trail at the end of the stub of road at the Rio Lado trailhead. This trail was supposed to be a circle on the end of about 2 miles of trail, but I found about 2 miles of trail and, unrelated to the location of the circle, some other random trails. There were even equestrians on one of them. Back to the mapping, I found that joining these various lines was difficult and I even managed to upload one without any tags at all. I went back and fixed things with iD, which isn’t appropriate. It does say that there’s a steep learning curve.

Then I moved on to the Calico National Recreation Trail. This is a motorcycle trail, but it actually does see plenty of hikers and mountain bikers, too. I was aiming at a bunch of peak bagging along its spine, but apparently was too rusty in my packing of my overnight backpack. I tagged Elliot Peak and returned. Then I took a different route up to the mountain spine to tag a few more peaks in an overnight. Sockrider first and the namesake Calico last.

… See full entry

This isn’t directly related to OSM, but many times in “software” or “development” OSM channels someone has asked “how do I do X … on Windows 10?”.

The reason that’s often a question is that, for historical reasons, much of the OSM software used to do things has been developed on Linux. That’s great for people working on Linux themselves, and the Apple Macintosh users can normally jury-rig something to run that software too.

Windows users can do the same too, by installing “Windows Subsystem for Linux”. Within Windows, search for “Windows Features”:

In Windows Features, tick the box for WSL and click OK:

… See full entry

Querying elements from osm and doing practical things with them is really a useful and fun way of code “hacking” for me. In this case, I wanted to show farmers markets with a reactive map like leaflet like I’ve done before in my journal entries. In this case, I wanted to make it easy to browse the times, seasons, and contact info for farmers markets in the Anchorage, Alaska area. I think with some refinement this map could be useful to sellers and buyers to determine which farmers markets they can fit into their schedules.

… See full entry

GIS data for listed buildings, scheduled monuments (e.g. interesting tumuli) and more can be downloaded from https://historicengland.org.uk/listing/the-list/open-data-hub/, and here are some observations based on my experiments with geoJSON and shapefiles.

The data is under the Open Government Licence so OK for editing OSM.

The download can be kept to a manageable size by enabling “Filter as map moves” in the filter menu (funnel symbol) and then “Toggle filters” when selecting the download.

Vespucci can display a geoJSON layer, show object attributes, and even create preliminary OSM objects from them. But it didn’t work here. I discovered that the downloaded geoJSON uses EPSG:3857 (projected coordinates) whereas the geoJSON standard is EPSG:4326 (latitude and longitude).

An easy way to do the conversion was to go to https://mapshaper.org, “upload” the file (actually it’s processed on the client side), open the console and enter:

-proj from=EPSG:3857 crs=EPSG:4326

and then export.

The resulting geoJSON works well in Vespucci. You can either “Add GeoJSON layer” from the layer menu, or select it in a file manager and open with Vespucci. I could do the whole process above on a modest android tablet.

OSM tags worth thinking about include:

• heritage = 2
• heritage:operator = Historic England
• ref:GB:nhle =
• listed_status =
• wikidata =

Note that I’m not suggesting adding anything to the map that can’t be located with a ground survey, and certainly not blindly copying polygons (aside from considerations of verifiable ground truth, these may include a protective buffer around the feature).

… See full entry

The problem with remote TIGER review (and solution!)

I recently did a major road alignment update for Rifle, Colorado. With JOSM and the to-do plugin, it only took a few hours and the road data is much improved. Doing the geometry check was time consuming but easy. Checking that the names are all correct? Isn’t that impossible from my desk? Thankfully no!

There are two kinds of name checking that I like to do. The first is to check that they are “sensible”, meaning that they have been expanded and there’s no obvious vandalism. The next is that they are “correct”. Because most road names originated from the TIGER import we shouldn’t use it to cross check for “correctness”. The good news is that the National Address Database has data for huge chunks of the country and provides us with another source to get a sense of how good the road network is named.

The workflow

… See full entry

It is very interesting to see the variations in data quality errors in OpenStreetMap per country (Asia Pacific Hub priority countries).

We broke down the type of error per country from OSMOSE and found out some interesting trends.

Like for example, it is noticeable that there are alot of overlapping building errors (~437,652) accounting for the 2nd most common mistake.

As map contributors, we should also put into highest standard the edits we add to OSM. Maybe in the next few months, we can host to bring these errors down.

Furthermore, some types of errors are generally acceptable like “building over agricultural landuse” and local language. Nonetheless, Thankful for Osmose for this powerful tool.

Link to Database

I’ve been updating OSM roads and trails based on the US Forest Service Motor Vehicle Use Maps (MVUM). The USFS MVUM is “the legal instrument that shows where people are allowed to drive”. There are many roads shown in OSM that are not open to motor vehicles, but not properly tagged. There are also few roads and quite a few moto trails missing.

If there’s a road that is not open I just add motor_vehicle=no. It’s possible that it could be access=no, but often it’s still open to bikes & almost always still open to foot or horse traffic. Since all I really know from the MVUM is motor vehicle access that’s all I usually add. I’ll update the ref tag if it needs it.

The data sources I’ve found for this are a nationwide SHP file and geopdf’s for individual forests.

The shapefiles are available here: https://data.fs.usda.gov/geodata/edw/datasets.php?xmlKeyword=mvum Since that’s the whole country, I use ogr2ogr -clipsrc bounds outfile infile [note the backwards order for the files] & clip to a manageable size. The whole state of Colorado seems to be a manageable size. I don’t directly import anything from the shp file, but If I see a fairly long trail or road missing I’ll copy and paste into a josm layer.

A web search of the forest name & MVUM will yield a page like this: https://www.fs.usda.gov/detail/psicc/maps-pubs/?cid=stelprdb5177824 (I just noticed that page has SHP files too.) I convert these to tif & load the image into JOSM. gdal_translate -co TFW=YES file.pdf file.tif JOSM is really sluggish with this so if someone has a better way I’m all ears.

My 1st step is usually to load the geopdf into Qmapshack overlayed with my custom OSM map built with mkgmap. I can fairly quickly see differences this way, easier than in JOSM. Once I know what part of the map needs work I load JOSM with the tiff image and make the OSM changes needed. I’m starting to use the SHP file more & may use the tif less.

… See full entry

Bing StreetSide is alternative to Google Street View which can be used for OpenStreetMap mapping. There is a MicrosoftStreetside plugin for JOSM but it doesn’t work for me due to dependencies of JavaFX/360 views.

I have managed to use it via Utilsplugin2.

1. Edit => Preferences (F12) => Utilsplugin2 settings
2. Add new row with name Bing StreetSide and url https://www.bing.com/maps?cp={#lat}%7E{#lon}&lvl=19.0&style=x.
   In the case of issues one can edit customurl.txt. For Linux/Flatpak it’s located at ~/.var/app/org.openstreetmap.josm/data/JOSM/plugins/utilsplugin2/customurl.txt.
3. Choose Bing StreetSide in Data => Select custom URL
4. Now you can see Bing StreetSide via Data => Open custom URL (Shift+H)

If StreetSide is available at given location it will be displayed. Otherwise map is shown with streets colored blue where it’s available.

What now?

In my previous diary entry I demonstrated using a Tableau visualization to find areas of the United States that could benefit from additional address data. If you find some place you’d like to work on, now what?

Getting set up (social)

OSM is a community project. It’s important to make sure mappers know what’s going and it’s always a good thing to give a heads up about plans for any big data changes in an area. Because this dataset is US centric I’ve been using the OSMUS Slack to keep people up to data about what’s happening. Each state has a “local” channel where you can get feedback and find folks to work with.

Always remember, not everything in the dataset needs to be added to OSM. It doesn’t cost anything to leave stuff out but can be quite time consuming to clean up if not done well.

Getting set up (tools)

Here’s what you’re going to need

… See full entry

I had been in this area, then run to lower elevations for a storm, then back for some trails I still wanted to do.

Work for mapping began with Hope Lake, where someone had managed to number the trail, but not name it. After the lake, they’d just marked it with a fixme. Yes, it’s the same trail. Then I got to playing with things over the hill and there was more of this very minimal editing to improve. Then I ran into the Colorado Trail. Um. The Colorado Trail is a mess, frankly. Someone decided it should all be named “Colorado Trail (Segment #)”. This is an area where the trail runs along older named trails. The Forest Service went hyphenating the name onto the old name on their maps to keep them both on the map, but it is two different names. Someone had copied it over, including keeping the Colorado Trail on a differently named trail after the Colorado had left it. And then I started running into the segment numbers. Why? Why why why? That’s a whole project in itself.

So I quit that and moved on. I thought about doing the Sheep Mountain trail, which may be informal but is well maintained including an astonishing amount of logging out the old road it follows. Unfortunately, I only joined it halfway along on my way down. I didn’t like the look of the mountain where it goes and took on some easy, if steep, mountain instead. The log at the top indicates Teluride is up here all the time in the summer. I dithered and ultimately did add what I could. There are complete tracks on Peakbagger, so I could potentially add all of it depending on the license there.

… See full entry

This is a crosspost from my blog.

I’ve been testing the latest alpha version of the RapiD editor. RapiD is a web-based editing environment for OpenStreetMap that incorporates special layers with map features from other sources you can easily copy over to OSM. Examples include Microsoft’s building footprint data, missing roads generated with machine learning, and open data from government GIS sources.

Because RapiD is not included in the dropdown menu where you can select an editor on the OpenStreetMap website, I created a few bookmarklets¹ for myself that I hope come in handy for mappers who want to have quick access to RapiD. They are self-contained and don’t read any content from your browser other than the current URL. If you’re not currently on openstreetmap.org or actively editing in the default OSM web editor iD, the bookmarklets will simply do nothing at all.

I tested these bookmarklets in Firefox and Chrome. You can find the source code here. When a new version of RapiD comes out, I’ll do my best to update the bookmarklets.

1. The bookmarklets are on a separate page, because this blog’s markdown parser has trouble with the javascript: links. ↩

I found an old mining road that’s being maintained as a trail while staying by the river, so I added that. Then I got all fiddly and added a bunch of driveways. I wish I’d taken a picture of the map BLM had on their information board at Caddis Flat Campground (added details about it) because that map had a more official trail a little further east, also leading to a mine.