OpenStreetMap
I started mapping in June 2020 as a way to find parks and trails near my home in Redmond, Washington. My daughter loves adventures and provides huge motivation. I am a former compiler engineer and bring a passion for great tooling. I love being able to work with others on an expansive, vivid, and important project.
This diary is a follow on to my previous entry detailing Adding addresses with JOSM and MapWithAI. You’ll need all of the setup there plus the Conflation Plugin.
The key to doing this quickly is to find an area with:
The area I’ve been spending most of my time recently is Phoenix, AZ which has great address data from the [National Address Database] and a high level of building coverage. The suburb are also quite sprawling which means you can cover a lot of very regularized ground very quickly. Here’s a good candidate for rapid addition: 
OSM power user SherbetS has been documenting the HIFLD dataset. It is a large corpus of public domain licensed geospatial information about infrastructure around the United States.
Download the dataset in GeoJSON format here. Fire up JOSM and open the file. This will create a layer that is just the HIFLD data. If won’t have any fields that look like OSM fields so DO NOT upload this directly.
Next we need to get the OSM data into JOSM. For this demo we will use the data for the state of Wyoming.
// fetch area “Wyoming” to search in
{{geocodeArea:Wyoming}}->.searchArea;
// gather results
(
// query part for: “amenity=hospital”
nwr["amenity"="hospital"](area.searchArea);
// query part for: “amenity=clinic”
nwr["amenity"="clinic"](area.searchArea);
);
// print results
out body;
>;
out skel qt;
We now have 2 layers. One with the HIFLD data and one with the OSM data.
We will now use the Conflation Plugin to match the nodes in the HIFLD dataset with OSM downloaded elements. Any elements that do not match to an OSM item should be reviewed and additions made. Any matched elements may be reviewed for completeness in OSM but that’s a separate matter.
We will start by selecting elements from the HIFLD dataset that are in our state of interest (“WY” in this case) and add this to the Conflation tool as the “Reference”. This is the set we’re trying to match elements to.
Did your JOSM UI text become a huge bother to read? Is it too tiny? Are the icons not as you’d expect? You may be suffering from a case of Too High DPI! It’s more common than you think!
Thankfully the medicine is easy enough to apply at home. You’ll need to find and edit the properties of the JOSM.exe file. It’ll be at a path like C:\Users\admin\AppData\Local\JOSM (you’ll need to change “admin” to whatever your user account is). Right click the exe > properties > Change high DPI settings > check “Use this setting to fix scaling problems…”
In the fall of 2002, there was a notice issued renaming many (~600) natural features in the United States. There was very likely to initiate a cascade of renames for various map features. For example, the name for the road up the mountain peak will likely change to match the new name.
At the tail end of our renaming work, I set up a few Google Alerts related to the theme of the order and promptly forgot about them.
Starting a few months ago the alerts began firing! Each week I would get an email or two about town councils or other agencies planning to rename some feature or another.
Another one came into my inbox this morning announcing that a resort near a renamed waterway was updating it’s name. A bit of checking the resort website to confirm the new name, a few clicks in iD, et voila!
It will certainly be interesting to see whether the roadway, golf course etc get similar updates and I’ve made a note to myself the check back in a few months.
This entry will describe one application of our GNIS matching program using a very narrow slice of the dataset.
While looking through the Populated Place feature class, I noticed a rather substantial number of names for mobile home parks. I was immediately reminded of the MapSwipe effort in coordination with YouthMappers and the ASU Knowledge Exchange for Resilience.
The full GNIS file has ~7000 entries that look like they will very likely be the location and name of a mobile home park. Running the GNIS Matcher against the ones in Arizona gives 516 places to check the map for either tagging improvement or new geometry.
You can check out the project here.
There is an amazing guide produced by ASU that has a huge amount of detail about mapping mobile homes. Definitely look it over.
An area drawn to represent a mobile home park should have tagging like:
landuse=residential
residential=trailer_park
name=Estrella Villa Mobile Home Park
There is occasionally some confusion between this and a similar tag tourism=caravan_site. This tagging may also show up in places with trailers/RVs but the intent is for that to mark temporary stopping locations, similar to tourism=camp_site.
The matcher would also like a GNIS ID included. The appropriate tagging looks like:
gnis:feature_id=2669824
In general, try to draw a boundary that encompasses all of the residences. There are often clues in aerial imagery that help delineate the specific park from other areas (guide). Look for fences, pavement differences, and road connectedness to guide your judgement. If there is high ambiguity, it may be helpful to leave a Note for local mappers.
Street side imagery from Mapillary or Bing Streetview is often helpful and has the appropriate license for OSM use.
If you’re new to this wonderful dataset I encourage you to read the OSM GNIS wiki entry or play around with the public GNIS web portal. Try feature ID 1629903!
The most important bits from a summary record are the feature Name, Class, and Coordinates.
Last fall, a group of mappers coordinated to address the name updates for Department of Interior Secretarial Order 3404. We cobbled together a collection of data scrapers and spreadsheets and declared victory a few weeks later. Since then, Kai and I have been thinking “there has to be a better way”.
For any entry in the GNIS National File we can ask the matcher to search OSM. It does this using a private Overpass instance and a set of heuristics about likely tag combination and feature types. With reasonably high fidelity we can:
We’ve generated a handful of MapRoulette challenges to check the functionality and see how it improves mapping. Here’s some examples that show off different feature types:
Sq___ Rename Validation - Final cleanup of rivers and other natural features listed in Department of Interior Secretarial Order 3404. It’s very good at noticing if a named waterway is mapped up the wrong tributary.
King County WA Place Update and San Diego County CA Place Update - Urban neighborhood names for the most part. This kind of task really does require local knowledge as the geo information for this class isn’t super precise.
Add and update Mobile Home Parks in AZ - Also from the “place” feature class but for an under-mapped feature type in OSM. This one ties nicely into a project run by MapSwipe and YouthMappers.
I started mapping in June 2020 as a way to find parks and trails near my home in Redmond, Washington. My daughter loves adventures and provides huge motivation. I am a software engineer by trade and from that bring my passion for great tools. I love being able to work with others on a project that is expansive, vivid, and important.
I recently did a major road alignment update for Rifle, Colorado. With JOSM and the to-do plugin, it only took a few hours and the road data is much improved. Doing the geometry check was time consuming but easy. Checking that the names are all correct? Isn’t that impossible from my desk? Thankfully no!
There are two kinds of name checking that I like to do. The first is to check that they are “sensible”, meaning that they have been expanded and there’s no obvious vandalism. The next is that they are “correct”. Because most road names originated from the TIGER import we shouldn’t use it to cross check for “correctness”. The good news is that the National Address Database has data for huge chunks of the country and provides us with another source to get a sense of how good the road network is named.
In my previous diary entry I demonstrated using a Tableau visualization to find areas of the United States that could benefit from additional address data. If you find some place you’d like to work on, now what?
OSM is a community project. It’s important to make sure mappers know what’s going and it’s always a good thing to give a heads up about plans for any big data changes in an area. Because this dataset is US centric I’ve been using the OSMUS Slack to keep people up to data about what’s happening. Each state has a “local” channel where you can get feedback and find folks to work with.
Always remember, not everything in the dataset needs to be added to OSM. It doesn’t cost anything to leave stuff out but can be quite time consuming to clean up if not done well.
Here’s what you’re going to need
After finishing my first address import I was looking for a good view of “where else needs addresses”. One trick that always pays dividends for me is to look at distributions of ratios of various quantities. In particular, I presumed there should be a pretty smooth distribution to the ratio of number of addresses vs number of buildings in any given area.
osmconvert alabama-latest.osm.pbf -o=alabama-nodes.csv --max-objects=50000000 --all-to-nodes --csv="@lat @lon addr:housenumber building"While working through some edits in Indonesia I noticed object with the key “nama”. A quick search revealed that this in Indonesian for “name” and the objects can very likely be modified to use the standard English. I wondered, how common is this? Is it easy enough to track down?
As a test run I picked 4 tags: name, building, source, type. These show up in TagInfo in abundance and I’m sure there are lots of other good candidates.
Next step is to get usable translations. It turns out Google Sheets has a GOOGLETRANSLATE function that takes a word and will return translations into various languages. I pulled in the two letter language code list and built my sheet. After eliminating all languages that Google Translate didn’t support and all languages with non Latin characters I was left with ~80 languages to check.
The last step was to pull usage information. Fortunately for me TagInfo has an exceptionally well documented REST API. Fifty lines of C# later and I had my results.
Clicking through a few of these in TagInfo reveals some more likely candidates for cleanup.
A common tagging mistake that I’ve encountered a few times is putting the addr:housenumber into the name field. Data of this type tends to be old so it seems the modern tools do a better job keeping this kind of edit from happening. However, when you do find this issue there’s usually a lot of objects to clean up which can be a bother without the right tools.
Here’s the overpass query I’ve used a few times that provides a reasonably stating place:
/*
This has been generated by the overpass-turbo wizard.
The original search was:
“"name"~"^[0-9]" and "building"”
*/
[out:json][timeout:25];
// gather results
(
// query part for: “name~/^[0-9]/ and building”
way["name"~"^[0-9]"]["building"]({{bbox}});
relation["name"~"^[0-9]"]["building"]({{bbox}});
);
// print results
out body;
>;
out skel qt;Update! User marczoutendijk from the Discord server has provided this overpass query that does a great job finding more instances if this with low noise:
(
node["name"~"^[0-9]+$"]["addr:housenumber"~"^[0-9]+$"]["brand"!~"."]["shop"!~"."]["amenity"!~"."]["highway"!~"."];
);The easiest workflow I’ve found it to load the area into JOSM and run a search with some more specificity. Here’s the search for any building way that has a name tag with numbers at the beginning and doesn’t have an addr:housenumber tag:
name~"^[0-9]+" and building=* and type:way and -"addr:housenumber"Or for names that are up to 3 numbers and nothing more (ex: 7,25,123):
name~"^[0-9]{1,3}" and building=* and type:way and -"addr:housenumber"Or a name that represents a range of housenumbers (ex: 12-16):
name~"^[0-9]+.[0-9]+" and building=* and type:way and -"addr:housenumber"Once you have the right set of objects selected, the tag editior will let you rename the “name” tag to “addr:housenumber”. I like to do a spotcheck or values in the name tag to make sure there’s nothing surprising and then the changes can be uploaded.