OpenStreetMap
I had a fantastic time at SOTM US. Thanks to all at OSM US for their work putting the event together.
I gave two presentations, one on MapRoulette and one on the new Meet Your Mappers tool. I posted annotated slides for the MapRoulette talk on my blog.
In the early days of OSM, when the map was formless and empty, mappers in the United States conducted data imports without much discussion, because there were just not a lot of people to discuss with.
One example is a series of imports of USGS GNIS data. This is point data from the United States Geological Survey’s Geographic Names Information System. Some of this data was very useful, for example to populate the map with place nodes for smaller towns. But there’s also a lot of data that was not very good, and a lot of it is still on the map today.
One example is the mines layer, imported with the tag gnis:feature_type=Mine. For starters, a lot of these nodes represent historic mines, of which the United States has many, especially in the West. But they were imported as landuse=quarry, a tag that should be used for nodes in the first place.
I organize the OpenStreetMap Salt Lake City monthly meetups. I don’t usually write reports on the individual meetups, but I had even more fun than usual last night so I thought I would write a quick report!
We had a great evening (as always) and with a good turnout too! We talked about SOTM US 2024, to be hosted in our fine city next June, and everyone signed up as a candidate volunteer for the event!
We also shared some knowledge about drones and related software; I know next to nothing about drones and haven’t kept up with the technology, so that was really interesting to me. We may pursue a small grant to acquire one for our OSM group. I am not sure from where yet.
As usual, we discussed recent business openings and closings to keep the map up to date. We have good resources like Gastronomic SLC (a weekly newsletter and website with news about restaurants and bars in the area) and City Weekly, a local newspaper, and of course our own observations.
Rapid 2.0 launches this week. The Rapid team will host webcasts on April 4 (tomorrow at the time of writing), April 5, and April 6 for Europe / Africa, the Americas, and Asia / Pacific timezones respectively. You can sign up here. You can expect an overview of what’s new, and a live demo. You will also be able to ask the Rapid team questions.
I wrote about the public beta of Rapid 2.0 before, and covered what’s new there.
One additional thing I wanted to call out is the ever-growing amount of external datasets available to mappers for efficient mapping of addresses, buildings and other features available as open data. There is a page on the OSM wiki that lists them all, and Esri has an interactive map with all the data sources available and considered as Rapid layers.
This is a cross-post from my blog.
This post is a follow-up on my series on GoPro Max panoramic imagery capture for Mapillary. Find part 1 here and part 2 here.
To capture true 360 degree images with a camera that has just two lenses, compromises are unavoidable. Optics dictate that a lens that captures a 180 degree field of view will have some image sharpness falloff at the edges of the field of view. I hadn’t considered this when I first started capturing with the GoPro MAX. I just mounted it the way I would a regular GoPro and didn’t give it another thought:
Until I started looking at the result more closely. Here’s a detail of a recent capture:
I wrote a long-ish post on my personal blog about how I used osm2pgsql to set up and maintain the database that powers Resto-Bot, the silly bot that toots out an OSM restaurant feature that has not been updated in more than 5 years.
This is a cross-post from my blog
I posted a couple of days ago about my first captures with the GoPro MAX camera I have on loan and uploading to Mapillary. This all went pretty smooth, so I wanted to capture while biking. I bike around town a fair amount and you can cover more ground in the same amount of time compared to walking, so I was eager to try it.
Years ago, I captured Mapillary imagery with my regular GoPro HERO 3+ camera mounted on my bike:
I remember from back then (2014) that the uploading process was not super straightforward, but Mapillary was only 1 year old then, so that was to be expected.
I still have some of the accessories from my HERO 3+, like a helmet attachment and a quick-release adapter. The accessory interface on the cameras hasn’t changed, so those things work even with the most recent GoPro cameras. Here’s the helmet attachment with quick-release shoe:
This is a cross-post from my blog
I have always been excited about the opportunities open street level imagery (SLI) offers to OSM. I helped launch OpenStreetView (now Kartaview) back when I worked at Telenav and was an early contributor to Mapillary as well. Recently, I’ve had the opportunity to work more closely with the great folks at Mapillary. This re-kindled my excitement for SLI & OSM! Good things are rolling out and will continue to roll out in 2023, so stay tuned! The Mapillary blog is the best source for announcements.
Most recently, Mapillary added support for 360° video upload to their desktop uploader and the CLI tools! This is great news for anyone with a supported camera, like the GoPro MAX. I happen to have one of those so time to try it out!
The Mapillary help pages include great and authoritative guides on using 360° cameras including specific instructions for using the GoPro MAX. This post is just me summarizing my own experience.
You only need to do these steps once.
I’ve been closely involved with the journey Rapid is on. The first version of Rapid was released in 2019 as the first OSM editor that added machine learning-derived layers you could easily add to OSM in one click: roads from Meta and building footprints from Microsoft. This groundbreaking work enabled efficient, but human powered adding of vetted external data to OSM, and continues to be one of the most widely used methods for doing so. Since the original launch, a collaboration with Esri’s community data program added many additional layers of authoritive data available to add to OSM in the same way.
As government agencies continue to make more data available to OSM through Rapid (around 145 as of now, with hundreds of millions of features), work on version 2 started in early 2022. Developers Ben Clark and Bryan Housel presented the early stages at State of the Map US in Tucson last March, and followed up with a more technical talk about the underlying technology changes at FOSS4G in Firenze. Alpha versions of version 2 have been circulating since then. I have been using these extensively, especially in the past two months. I’ve been barraging Ben and Bryan with bugs and feature requests, as have many others who have participated in the alpha testing phase. A ton of issues have been resolved since the first alpha release, and we’re very happy with where we are with Rapid 2.0. Happy enough to launch version 2 beta today! 🎉
If you haven’t used Rapid at all yet, this is a great opportunity to try it for the first time. Many things will feel familiar if you’ve used the built-in editor on openstreetmap.org, but you will also notice small conveniences you won’t find there—or anywhere—like a shortcut to quickly cycle through highway types when you have a way selected, “virtual nodes” that are displayed for polygons with POI tags and improved polygon labeling:
This is a crosspost from my blog.
I’ve been testing the latest alpha version of the RapiD editor. RapiD is a web-based editing environment for OpenStreetMap that incorporates special layers with map features from other sources you can easily copy over to OSM. Examples include Microsoft’s building footprint data, missing roads generated with machine learning, and open data from government GIS sources.
Because RapiD is not included in the dropdown menu where you can select an editor on the OpenStreetMap website, I created a few bookmarklets1 for myself that I hope come in handy for mappers who want to have quick access to RapiD. They are self-contained and don’t read any content from your browser other than the current URL. If you’re not currently on openstreetmap.org or actively editing in the default OSM web editor iD, the bookmarklets will simply do nothing at all.
I tested these bookmarklets in Firefox and Chrome. You can find the source code here. When a new version of RapiD comes out, I’ll do my best to update the bookmarklets.
The bookmarklets are on a separate page, because this blog’s markdown parser has trouble with the javascript: links. ↩
After tinkering a bit but finally successfully processing videos from my Viofo A129 dash cam and uploading them to Mapillary, there is one last thing I wanted to try: using the A129 time lapse mode to be able to collect more imagery before the storage space on my camera runs out…
This does not seem to be possible without recording your GPS breadcrumbs using a separate device, because the location information written into the movie stream by the camera is sparser than the video frames. Using
exiftool -m -p gpx.fmt -ee -ext mp4 -w! %f.gpx time-lapse-movie.MP4
I get 4 trackpoints for a movie that contains about 300 frames.
This could also be a result of limitations in the way exiftool parses the MP4 file, but looking at the relevant documentation section I don’t see a way to tweak this.
What I think I will do instead is:
In order to be able to capture more of my longer trips. I’m about to make [this drive] and I’ll test it then!
A few days ago I wrote about extracting GPX location data from the raw videos coming out of my A129 dashcam, and uploading to Mapillary. I was doing this one video at a time. A typical drive yields a lot of short videos, each 1, 5 or 10 minutes long depending on the settings. Some automation would be nice!
Mapper n76 suggested in the comments to my previous post to concatenate the short videos first and then process the resulting single video. I tried this following the instructions on the ffmpeg website, but I could not get exiftool to extract location data from the resulting longer video. So what I did instead was write a simple bash script that just loops over all MP4 files in the directory and does the GPX extraction and Mapillary processing / uploading for each file. Here’s the full script I used, which has the gpx.fmt file you need for exiftool baked in for convenience, but the loop itself is simply:
for f in *.MP4; do
exiftool -m -p gpx.fmt -ee -ext mp4 -w %f.gpx $f
mapillary_tools video_process_and_upload $f --geotag_source gpx --geotag_source_path ${f%%.*}.gpx --skip_process_errors
done
I found that I need --skip_process_errors because there’s usually one image extracted from the start or end of each video file that cannot be matched with a timestamp from the GPX file. I don’t care enough about one single image out of an entire sequence to try and figure out why, but I’m sure someone more determined than I could fix it :)
I purchased a dashcam a while ago because (1) people drive like absolute idiots where I live (seriously, watch that video) and (2) who knows what you might capture? It’s a Viofo A129 Pro Duo, it has a 4k front-facing camera and an additional HD rear-facing camera. It also has built-in GPS.
Because it’s always on, I figure it would be nice to use the footage for mapping purposes as well! The process is not completely straightforward, hence this blog post.
Let’s look at what the camera produces:
I recently wrote about using a Tag-Fix MapRoulette Challenge to update OSM data in an efficient way, no editor required. Tag-Fix is one of two newer Challenge types MapRoulette has available. The other one is called Cooperative. They are called Cooperative because it’s a cooperation between you and the Challenge owner. The Challenge Owner provides the OSM changes, you validate them, tweak them as needed, and commit them to OSM. Cooperative challenges are ideal for when you may be thinking about an import, but you would like to have each feature manually verified by a mapper.
Today, we will look at how to set up a Cooperative Challenge using Microsoft Building Footprints open data. (In a future post we’ll look at how to work with the Challenge as a mapper.) This is not really a beginner’s tutorial; we will be working with PostGIS, QGIS and the command line tools imposm and ogr2ogr. If you have those tools installed and are just a little familiar with them, you should be able to follow along fine!
This is what the final result will look like:
I’ve been working with the Mapillary team through my job at Kaart. In the process, I’ve been stepping up my Mapillary contributions quite a bit. Here’s a recent capture on the way back from an afternoon of skiing!
source: Mapillary
A recent-ish new feature (on iOS only for now) is Mapillary Missions. Missions focus on specific areas where there’s particular benefit to OSM in capturing new(er) imagery, for example because there’s potentially high POI density, or the existing images are stale. Individual missions are small, usually around 300-400ft along one street. Here’s a few in my area:
I am terrible at keeping journals, and this will probably be no exception, but I have been enjoying some folks’ regular updates about what they have been up to in the OpenStreetMap world. So let’s give this a try! Two weeks late already for October; life got in the way..
I’ve had a pretty active mapping month, but also all over the place, more so than usual.
Americans love cars. More than 90% of households own one, more than 20% of households own 3 or more. Cars stand still most of the time and for that, we need huge amounts of parking.
Image source: Flickr Commons
The simplest way to map a parking area in OSM is to draw an area and mark it amenity=parking. It will then show up on the map as a grey area with a blue “P”. In the United States, almost a million areas exist with the amenity=parking tag.
I’ve spent a few evenings and spare hours this week dusting off and updating osmdiff. If you’re a python developer and you need to interact with replication diffs or augmented diffs, this may just be something of interest to you. osmdiff can retrieve replication diffs as well as augmented diffs from OSM servers, and parse them into native Python Node, Way and Relation objects you can use in your code. This lets you do things like
>>> from osmdiff import OSMChange
>>> o = OSMChange()
>>> o.frequency = "minute" # the default
>>> o.get_state() # retrieve current sequence ID
>>> o.sequence_number
2704451
>>> o.retrieve() # retrieve from API
>>> o
OSMChange (677 created, 204 modified, 14 deleted)
Once you have the data, you can filter and inspect it:
>>> w = [n["new"] for n in a.modify if n["new"].attribs["id"] == "452218081"]
>>> w
[Way 452218081 (10 nodes)]
>>> w[0]
Way 452218081 (10 nodes)
>>> w[0].tags
{'highway': 'residential'}
>>> w[0].attribs
{'id': '452218081', 'version': '2', 'timestamp': '2017-11-10T13:52:01Z', 'changeset': '53667190', 'uid': '2352517', 'user': 'carths81'}
>>> w[0].attribs
{'id': '452218081', 'version': '2', 'timestamp': '2017-11-10T13:52:01Z', 'changeset': '53667190', 'uid': '2352517', 'user': 'carths81'}
>>> w[0].bounds
['12.8932677', '43.3575917', '12.8948117', '43.3585947']
You can also use osmdiffs implementation of Node, Way and Relation objects separately if you need just those, the library is quite small and has few dependencies (just requests and dateutil)
On the list of things in development and slated for the next release are:
* Python __geo_interface__ support so you can easily use them in other Python modules that support them like shapely and geojson
* Retrieving individual OSM features from the OSM API
* Ability to create an AugmentedDiff / OSMChange object with a datetime parameter, which will then calculate the sequence number for you
* More tests, always more tests, and CircleCI integration
After I reconnected with Ilya at SOTM and talked to him about his new app Every Door, I thought it would be nice to organize a mapping party around it back home. I just got back from some Mall Mapping with a small OSM Utah group, and wanted to share my experiences with the app.
First things first, the app works great. The fact that it’s on Android and iOS and looks and works the same on both platforms is great. The interface is snappy, there’s no annoying crashes or delays, and everything is fairly easy to discover and learn how to use.(However, this is coming from a group of people who have experience with OSM mapping and are at least a little savvy about technology..)
There were two interface elements that took us a little time to figure out. One was the “modes” at the bottom. The default mode is POIs (the coffee cup icon). It is not immediately clear what the other modes do, but for people with some experience with OSM, you can figure it out in a few minutes.
From time to time, I come across mappers that really specialize on a specific mapping topic. I was doing some random mapping in my state when I came across this very well mapped school:
This mapper, Chef7, has mapped lots of schools across the United States in high detail: