IoU is still below 0.8

If you reach an IoU of 0.8 that’s pretty amazing to be honest. Here’s why. There are two sides contributing to the IoU metric: your predictions can be off but worse also the OpenStreetMap “ground truth” geometries can be off. Even with a perfect model you won’t reach an IoU of 1.0 since the OpenStreetMap geometries can - and often are - coarse, or have a slight misalignment, or are not yet mapped in OpenStreetMap etc.

Here’s an interesting experiment: randomly sample from your dataset. Manually annotate the image tiles generating fine-detailed masks. Now calculate the IoU metric on your manually generated fine-detailed masks aand the automatically generated masks from OpenStreetMap. This will be your IoU upper bound you can reach.

Also see this graphic to get a feel for the IoU metric.

Training this without GPU takes almost a week :-)

Agree, without a GPU training will be slow. That said, I made some changes recently which should speed things up considerably:

• https://github.com/mapbox/robosat/pull/65 - Simplifies the training setup; now you no longer have to manually tune learning rate and more importantly epochs at which to decay the learning rate. This should give you great training results without any tuning.

• https://github.com/mapbox/robosat/pull/46 - We are using an encoder-decoder architecture. Before we were training both the encoder as well as the decoder from scratch. This changeset brings in a pre-trained ResNet encoder, resulting in faster training times per epoch, less epochs needed, less memory needed, higher validation metrics, and faster prediction runtime.

If you want to give it a try with current master you should see improvements for your use-case.

The amount of images you will need for training can vary a lot and mostly depends on

• the imagery quality, and if it’s from the same source or not
• how good and precise the masks for training are
• the zoom level
• the variety in the images, and if it’s from the same area or totally different
• the time and processing you can invest

For example the more hard-negative iterations you do the better the model can distinguish the background class. But hard-negative mining also takes quite a while. Same with the automatically created dataset: you can manually clean it up it but it is quite time-intensive.

In addition you could do more data augmentation during training to further artificially embiggen the dataset, you could do test-time augmentation where you predict on the tile and it’s 90/180/270 degree rotations and then merge the predictions, you could train and predict on multiple zoom levels, and so on.

I would say it also depends on your use-case. For detecting building footprints like in this guide a couple of thousand image are fine to get the rough shapes. It’s definitely not great for automatic mapping but that is not my intention in the first place.

Regarding trained models: I recently added an ONNX model exporter to RoboSat which allows for portable model files folks can use with their backend of choice. I could publish the trained ONNX model for this guide since I did it on my own time. The Mapbox models I am not allowed to publish as of writing this.

If there is community interest maybe we can come up with a publicly available model catalogue hosting ONNX models and metadata where folks can easily upload and download models?

Can you create an issue in the https://github.com/mapbox/robosat repository. I haven’t seen rs extract erroring out; there may be some warnings due to invalid polygon shapes but that should be it. Let’s follow up on a ticket.

@Rory: here are some rough numbers: with a single GPU you can expect training on ~20k images to take in the order of multiple days. Prediction on a GPU will take between 350 ms – 750 ms per 512x512 image tile - depending on how much overlap you buffer per tile to handle tile boundaries. Doing the same on CPUs and especially your laptop will probably slow things down by a factor between 10x – 50x at least.

What you can do is prepare all the datasets and get ready for training. Then spawn up a AWS p2 instance (install nvidia drivers, cuda, cudnn) and train on there for a few days. Save the model, shut down the instance, predict on CPUs. You could also look into getting a GTX 1080 TI but it’s only really worth it if you want to do these sort of things on a large scale for a couple of months at a time running 24/7.

There are “bottlenecks” in the road network which the partitioner detects. You can think of it as answering the question: which roads do I have to cut out of the road network to split it in half. Then we optimize for the minimum amount of roads to cut out and we balance the cuts to divide the road network “roughly” in half on every recursion level. This gives us roads we know cars have to go through when driving from one cell to the other.

Here are two maps from almost three years ago when I prototyped the partitioner and did these visualizations at a Geofabrik hack weekend in Karlsruhe. I remember we were wondering if the partitioner would cut old east and west Berlin in half.

The first one shows multiple cuts for Berlin. We run multiple cuts with a different slope in parallel and take the best cut on every level. Then we recurse down repeating the process in parallel for both sides.

Map - Berlin

And a single cut how to split California in half removing the minimum amount of roads.

Map - California

Note: this was from a prototype and probably does not match what you will currently get from our actual implementation in the Open Source Routing Machine.

Here is the current implementation:

• https://github.com/Project-OSRM/osrm-backend/tree/master/include/partitioner
• https://github.com/Project-OSRM/osrm-backend/tree/master/src/partitioner

You can definitely make use of these partitions for visualization or other purposes.

Also check out the Inertial Flow paper.

Thanks, I just added a note to the original diary post.

Die neuen Resultate koennen hier gefunden werden:

Up-to-date results can be found here:

We still don’t have regular updates or statistics mostly because I’m doing this every now and then on the side when there’s time for it and we still don’t have all checks we want to have in place. If someone wants to help out here I’m happy to get you started.

Trying something new this time: showing a line string for the from and to segment. It’s the routing engine’s normalized graph in the background so the linestring does not follow the way geometry. Hope that still helps to figure out the turn more easily.

You can follow the discussion in the routing engine’s ticket here.

We refined some checks and added new ones for sharp turns (e.g. sharp turns from one ramp onto another). Here are up-to-date results for the planet:

https://api.mapbox.com/styles/v1/danieljh/cj4zptszf0xgl2srvbsseiwjw.html?access_token=pk.eyJ1IjoiZGFuaWVsamgiLCJhIjoiTnNYb25JSSJ9.vYOnsuu1zeKcGW2nj0uJZw#6.22/51.134/11.663

Colored as follows:

• < 45 degree #e55e5e (red)
• < 65 degree #f9886c (orange)
• < 85 degree #fbb03b (yellow)

Wir haben die Checks etwas verbessert und neue hinzugefuegt (z.b. fuer spitze Winkel zwischen Ausfahrten). Resultate fuer den Planeten sind hier:

https://api.mapbox.com/styles/v1/danieljh/cj4zptszf0xgl2srvbsseiwjw.html?access_token=pk.eyJ1IjoiZGFuaWVsamgiLCJhIjoiTnNYb25JSSJ9.vYOnsuu1zeKcGW2nj0uJZw#6.22/51.134/11.663

Farben wie folgt:

• < 45 degree #e55e5e (Rot)
• < 65 degree #f9886c (Orange)
• < 85 degree #fbb03b (Gelb)

Where there is NO place for your scenario. Even thought the turn to motorway link might be < 90 deg for one of the directions (it will be unless is it exactly 90). There is no reason to think it is a mistake. You should at least check the onway tag.

We do check the oneway tag - not even that but we check hopefully most if not all routing related tags; if you find a tag missing please open an issue in

https://github.com/Project-OSRM/osrm-backend/issues.

The location you pointed out is problematic mostly because of this turn:

http://map.project-osrm.org/?z=18&center=49.723316%2C13.355816&loc=49.723387%2C13.356231&loc=49.723484%2C13.355427&hl=en&alt=0

If you have a look at how the intersection is modeled in OpenStreetMap:

osm.org/edit?way=139088393#map=19/49.72327/13.35573

and now compare it to reality by looking at the satellite imagery.

In this case all the routing engine sees from the data is a sharp turn onto a ramp; that the intersection here should be modeled out more detailed is something the routing engine can not tell.

I will run the same validation handler and a new one on the planet tomorrow or the day after and will post back with new GeoJSON data and a map.

It would be great to find false positive categories. For example some of the locations I samples had a stop sign missing and were actual legal turns.

If there are false positives which we can detect from the OpenSTreetMap data I’m more than happy to implement this in OSRM.

Interessanter Sonderfall! Der Router sollte den Serviceweg in der Realitaet nie bevorzugen, da er entweder gar nicht im Graphen ist oder langsamer als die Auffahrt ist.

Ich wuesste aus dem Stand jetzt auch nicht ob der change/change:lanes Tag an der Stelle angebracht ist, oder wie das genau zu taggen ist. Falls du das rausfinden kannst waere ich sehr interessiert daran!

OSRM versteht die lane change tags (noch) nicht und die Validation ueberprueft wirklich nur Winkel: https://github.com/Project-OSRM/osrm-backend/issues/3294

Als langfristiges Ziel haetten wir gerne einen (optionalen) road-network validation pass in osrm der dann GeoJSON mit entsprechenden properties erzeugt. Das ist im Moment allerdings noch etwas zu frueh um sowas vollstaendig zu automatisieren, vor allem wenn es viele false-positives gibt.

You can also find the raw GeoJSON for the initial run on the planet in this ticket:

https://github.com/Project-OSRM/osrm-backend/pull/4160#issuecomment-309253836

it will probably change with time when refining the check and adding more features to it.

I think Osmose has some destination checks. Not sure what kind of specific QA you want to do on destination tags, though. Care to elaborate?

The Open Source Routing Machine shows you these connectivity issues, as we are already running Tarjan’s algorithm for strongly connected components on the graph.

• On the demo server bottom left “Small Components Layer”
• In the debug map in addition to maxspeeds and turn penalties

both times in pink.

The same layer is getting published to the OSM Inspector, too.

I have no local knowledge of the roundabout in Towson - from what I can see the traffic inside the roundabout always has the right of way (needs to be confirmed). Therefore a proper junction=roundabout tag is appropriate. Multiple lanes is not an issue here.

Ja, theoretisch gibt es bei Fähren auch die Unterscheidung zwischen

• Kann der Benutzer dort warten, z.b. weil die Fähre alle 5-10 Minuten abfährt / anlegt
• Muss der Benutzer die genaue Abfahrtszeiten wissen damit es Sinn macht dort hingeleitet zu werden

Der Unterschied bei Fähren ist dann noch, dass du oft keine Alternativen hast bzw. nur einen immensen Umweg fahren müsstest. Außderdem sind Fähren zum Glück weitgehend mit dem duration Tag versehen, der für die Routenplanung geparst werden kann.

Baustellenampeln sind lustigerweise von der Charakteristik her nicht von oneway=alternating zu Unterscheiden. Aber temporär ist immer so eine Sache ..

The distinction should be between

• Does it make sense for the user to wait there for the direction to flip
• Does it make sense for the user to go there without knowing the exact point in time

As mentioned, it would be great to have frequency / interval tags on all of them. Then the routing engines could decide on their own and use the data e.g. as penalty or for time-dependant routing. Of course dynamic mechanisms like sensors make this harder.

What we really want to accomplish with oneway=alternating is the clear distinction from oneway=reversible’s time-dependence.