It has been a few weeks since I wrote about the public beta release of Cygnus, the Telenav conflation engine for OSM data. Since then, I have since been approached by a few folks who wanted to take it for a spin. One of them is long time OSM contributor MikeN. He is preparing an import for Holt and Atchison counties in the U.S. state of Missouri. We worked together on scaling some technical hurdles. Here’s a report of what we (well mostly he) did.

Source Data

Mike obtained the source data from Holt and Atchison counties from their official GIS:

I obtained an updated road network from their official GIS, extracted and translated the tags, and followed up with a review against current aerials as well as checking for connectivity, glomming like road segments, and simplifying geometry. The final goal is to obtain permission to import, go through the import process steps, and merge new data onto the existing OSM data.

The next step was to convert the data into the OSM PBF format that Cygnus requires. This is when Mike got in touch with me to work through some technical difficulties:

Since Cygnus required the PBF format, I used Osmosis to convert. This failed because the nodes did not “have a version attribute as OSM 0.6 are required to have”. I have learned from Martijn that OsmConvert works without a version attribute, and was able to verify this on my second county.

The next catch was that PBF doesn’t accept negative node numbers. The simple workaround is to just use a text editor to remove the minus sign from id='- and ref='- . This seems a bit dangerous - would that file upload if accidentally selected? If so, many low numbered objects would be corrupted around the world. Hopefully, the conversion from OSM to PBF can be moved to the Cygnus chain so that it can accept zipped OSM since most users will start with .OSM data.

… See full entry

I am happy to announce that Improve OSM, the suite of open source Telenav tools that help us fix OSM based on vast amounts of GPS data, is completely redesigned and expanded. We made the web site much easier to use. We combined the existing JOSM plugins into one new plugin. And as of today, we have an entirely new category of fixes: missing turn restrictions. I will talk about all of these changes in this diary entry. A lot to cover so let’s get started!

The new Improve OSM web site

We redesigned the Improve OSM web site completely. Instead of separate maps for each type of error, all errors are now displayed on one map. You can turn each type on and off in the layer panel. You can also control the filters for each type there.

What’s even better (I think) is that you can now perform the entire fixing workflow from the web tool, including resolving the error as fixed or invalid. (You used to need the JOSM plugin to change the status on items.)

… See full entry

Missing Roads is doing great! Since we launched the plugin a little over two months ago, almost 14 thousand Missing Road tiles were closed.

A little over a quarter of those were marked as invalid. We have been looking carefully at those cases, and it turned out the two most occurring invalid tiles were ones involving foot / bike paths and water. So we created filters for those. This should help you use the Missing Roads plugin much more efficiently!

Here is the filter in action for water tiles in the web tool:

… See full entry

Update After talking to a lot of people about the future of MapRoulette and getting a lot of great feedback, I decided to abandon the Django effort I wrote about here. The New MapRoulette is coming together nicely! See its progress on Github.

I recently started looking into Django, the seminal Python web application framework. I like what I see much more that I thought I would, so much so that I am starting to rewrite MapRoulette as a Django application.

One thing I want to accomplish with a next version of MapRoulette is to make creating challenges much, much easier. For example, you should be able to define a challenge using an Overpass query. (You would be surprised how many annoying errors in OSM can be exposed with a pretty simple Overpass query! For example, here is a query that gives you all highway=tertiary that have no name in my home state Utah - 425 ways.)

So I needed a way for a Django application to access the Overpass API and display and store the results. That should be simple enough to accomplish using the Overpass API Python wrapper project I started a while ago. And it was!

… See full entry

I wrote about Cygnus, our effort to create an OSM-specific conflation engine, a few months ago. We developed it specifically to aid the import of INEGI road data in Mexico we are preparing together with the community in Mexico. But when used with care, I think it can be very useful in other import scenarios as well. This is why we decided to make it into a web tool anyone can use.

Before I go into details, I should offer a few words of caution.

Firstly, this tool is an early beta. Right now, it only conflates roads, nothing else. We have tested the tool internally, but only on a limited amount of cases, all using open INEGI data from Mexico. With this public beta, I hope to gather more feedback to help us make improvements to it.

… See full entry

You can be more than one kind of geek at the same time, I guess. First and foremost, I am a map geek. This started when I got my first atlas for my 6th birthday. Then came computers, with the arrival of the Commodore 64 I shared with my brothers. Shortly after that came trains. I still have a few thousand slides of stations, trains and railyards in my basement. Also, German and Dutch timetables dating back to 1985.

One of my later railway photos. Berlin Lehrter Stadtbahnhof, summer 1996. This was demolished not too long after to make room for Berlin HBf. Source: Flickr

… See full entry

I announced our second tool based on Scout GPS data, Traffic Flow Direction, a few days ago. I didn’t spend a lot of time on explaining how it works. This post will hopefully make up for that!

The goal of this plugin, and the accompanying web tool, is to make it easy to find and correct OSM ways that we think are missing a oneway tag, based on billions of GPS points from Scout users. Here is what it looks like in JOSM:

I will walk you through installation, basic operation and some mapping tips in the next paragraphs. Happy mapping!

Installation

This is a JOSM plugin so, installation works like any JOSM plugin. Make sure your JOSM is up to date first. Then go to JOSM preferences. Select the Plugins tab and look for the TrafficFlowDirection entry:

… See full entry

Last month, we released the Missing Roads plugin and web tool. This quickly became a popular pastime for quite a few mappers - one month later, more than 20% of all the Missing Road tiles have already been resolved :)

For the darker and colder November days – well, if you are on the Northern hemisphere at least – we thought we would cook up something new to keep us all busy. We ran another analysis on our GPS data to uncover ways that probably do not have the right directionality. Either they should be oneway=yes and they are not, or they are oneway=yes but in the wrong direction.

Here is an example from Karlsruhe, Germany:

The orange arrow points in the direction we think traffic on that street flows based on what we know from our Scout GPS data.

… See full entry

I have long wanted to try and do some audio mapping. Especially since I moved to the US and started spending more time in a car.

When you are driving, there is not a lot of ways you can record what you see in a way that makes it easy to map later. One way is to use Mapillary, but the sheer amount of information can be overwhelming. A picture every 5 seconds means 180 images to go through on a short, 15 minute drive. It also means handling over 300 MB of image data. And that’s only for a 15 minute drive.

So audio mapping. I have this tiny recorder that weighs almost nothing, has built in space for almost 70 hours (!) of recording and runs weeks on a set of AAA batteries:

… See full entry

A few weeks ago, I started something new for the Salt Lake City area OpenStreetMap group: Flash Map Mobs! The idea is to have a group of mappers descend on an under-mapped area and add lots of shops, restaurants etc. in a short amount of time. We do them after work, say from 5-6. Sometimes we get a drink or dinner afterwards.

We have done four so far and I wanted to share some results.

Flash Map Mob 1 - Sugar House

• Meeting Location: Sugar House Coffee - meetup
• Attendees: 3
• Focus: 21st South / Highland Dr area
• POIs added: 88 - Overpass Turbo

The first one! I decided to meet at Sugar House Coffee because I go there a lot. This place can get a bit crowded and loud however. This may turn people off so perhaps not perfect as a meeting place. The Mob itself (then still called ‘After Work Map&Meet’) was fun. We shared experiences and arrived at Pushpin OSM as the ideal iOS mobile editor for this kind of mapping.

Flash Map Mob 2 - Holladay

… See full entry

Last week, we released the Missing Roads JOSM plugin, together with a web tool, the plugin source code, and a manual. More than 700 mappers have already given it a try, and nearly 10% of the missing roads tiles are already resolved. That is fantastic progress!

We have received a lot of great feedback from you and we are looking into every single suggestion. You may have noticed that we have already updated the web app adding a location hash. This means you can now share the URL at any time and it will include the current map view.

We also received a few requests to get access to the data powering the plugin and app. This sounded like a great idea to us! We started publishing daily dumps of the tile data. You can download and use these any way you see fit.

This is what the data looks like:

… See full entry

The most fun I have with OSM is making the map better. Adding nodes and ways and tags. Seeing how my puttering around with JOSM or iD leads to a prettier map that is more useful.

Getting folks together in Salt Lake City where I live to have some OSM fun together. For example. A few weeks ago, I started a new thing called the Flash Map Mob. We do them every other week now in SLC and together we map something like 100 shops and restaurants every time.

The map becomes more useful and folks are having fun. That is what OSM is to me!

I don’t need to sit on any board to do these things. Heck, I can probably do more of them when I am not on the board! So why do I do this? What value do I add being on the OSM US Chapter board?

I’ll tell you and I’ll be brief.

OSM is still very good at adding contributors.

(The graphs come from osmstats.)

What we’re not so good at is actually getting folks to go out and map. Look at the flat daily active mappers graph for the US:

… See full entry

Earlier this week, we released the Missing Roads project. Using its tools, every OSM mapper can now easily find roads that are not yet in OSM, based on our years of collected global GPS data from Scout users.

A lot of you have already had some fun with the JOSM plugin and the web tool. I received interesting reports about important roads you have been able to add. But also some questions about how to optimally make use of the Missing Roads tools. So I thought I might write up a Missing Roads Manual of sorts.

Missing Roads consists of two tools: a web tool and a JOSM plugin.

The Web Tool

The web tool is a convenient way to locate missing roads in an area. You can quickly get a sense of the distribution of the missing roads data.

If you zoom in far enough, you can also see the individual tiles.

Colors of tiles and traces

You will notice that there are different colors for both the traces and the tiles themselves.

… See full entry

New roads are built and opened for traffic around the world every single day. In many places, these are added to OSM by watchful mappers right away. Not everywhere though. There are still many places where there are few local mappers, and new construction goes unseen for a while. Available aerial imagery can be pretty outdated, so armchair mappers are not always able to help out either.

Lots of people drive on these new roads from day 1. And we’re in luck - a bunch of them are usually Scout users :) This means that they contribute GPS traces of the missing roads to us. Lots. Of. Traces. Enough for us to come up with a good guess about where new roads might be that are not on OSM yet.

We have done just that! I am excited to announce the Missing Roads project, opening up the aggregated, processed GPS data pointing at missing roads as a convenient JOSM plugin.

… See full entry

K1wi’s recent diary showed us how the use of destination is catching up with exit_to in the US for exit sign tagging:

I am excited about this! You may recall that I am very much in favor of destination on the _link way to tag exit signs rather than exit_to on the junction node.

I’ve come up with an idea to squash the remaining exit_to-tagged nodes in the US and move the exit sign info to a destination tag.

No scripting. No funny stuff.

A MapRoulette Challenge!

This is what it would look like:

… See full entry

Looking for new ways to get people out to map, I started a new initiative in our local Salt Lake City OpenStreetMap group: The Flash Map Mob. Inspired by the Flash Mob, the idea is to descend on a local commercial area, spread out and map all the businesses in an hour or less. We tyically meet at a coffee shop, divide up the area, and go out and map! Most people use their smart phones or tablets with apps like Pushpin, Vespucci or OSMAnd. This way, everything you map gets added to OSM right away and there is no work to be done after. But you could also use pen and paper, or GPS + camera.

Here we are checking out Pushpin on an iPad:

Here are the results of the first Flash Map Mob we did a few weeks ago, where we mapped 80+ businesses with three people:

… See full entry

I don’t do a lot of ‘on the go’ mapping. I have tried a lot of the tools out there that should help me do that: KeypadMapper, Go Map!!, Vespucci, Pushpin, OsmAnd. Most of these are way too convoluted for me. When I am out and about, I just want to make a quick edit - usually a POI of sorts - and move on.

Pushpin and Go Map!! come the closest to what I want. Both are available for iOS only, I think. (Do Android users not like simple editing tools?) They let me quickly add a node and upload it to OSM. Or add some tags to an existing one.

Still, what I end up doing most is just taking pictures with my phone. Lots of them. I try to get both overview and detail. I end up with a collection of images like this one here from right before I moved to the US.

When I get home, I copy all the images to my computer and load the entire directory into JOSM. It will put the image locations on a map. I can cycle through them easily and map what’s on there using all the convenience of JOSM.

… See full entry

OpenStreetMap in 2007 and today…

Mexico released a huge amount of open data not too long ago.

Huge: Mexico's statistical institute INEGI goes open data @INEGI_INFORMA (via @rodowi — Alex Barth (@lxbarth)

A lot of this data is geospatial, so I say yummie! Alex Barth wrote about this data, that comes from the Mexican national statistical agency INEGI, on his diary before, with a nifty map to show how rich this data is:

(My mediocre animated GIF skills really don’t do it justice - check out Alex’s blog post to see an interactive map.)

So now the question becomes: how do we get some (or all?) of this data into OSM? This is not straightforward - OSM already has rich data in many places in Mexico we would definitely want to keep.

Here at the Telenav OSM team, we have come up with an answer to this question. We call it Cygnus - The Bringer of Balance. Let me explain in a few visuals what Cygnus does.

Consider this area in the Aguascalientes region. There is some OSM data there:

… See full entry

UPDATE We are working on getting the source code out as Open Source as well. I will post with more details when I have them.

ScoutSigns is a JOSM plugin developed by my colleagues here at Telenav that displays speed limit signs (and a few other types) as a layer in JOSM. These signs are detected by users of our global Scout app if they opted in and have their phone mounted on the windshield. This is a pretty good source of sign information ready to be mapped!

JOSM showing Scout detected signs in ScoutSigns in Hamburg, Germany

Recently, Mapillary started detecting all kinds of signs in their millions of user-uploaded street view images. What would make more sense than collaborating with them to display their detected traffic signs right along with ours in Scout Signs? That’s exactly what we did!

… See full entry