After many years, I finally decided to dive back into running a full planet import using the Overpass API.

To keep things budget-friendly, I rented a modest Intel-based server for around 40€/month — nothing fancy, just dual data center SSDs and 64GB of ECC RAM.

What surprised me was the runtime: the initial import using release 0.7.58.5 took a whopping 33.5 hours — at least 10 hours longer than I had anticipated. I managed to shave that down to 26.5 hours by tweaking some settings, like enabling LZ4 compression across the board and increasing the chunk size parameter. It helped, but clearly, there’s still room for optimization.

I continued testing with my own experimental Overpass fork, that includes support for PBF, multithreading, and many other changes under the hood. Initial measurements looked quite promising with 10.5 hours total runtime. After some further analysis and improving some data structures, the import took 7 hours and 23 minutes. Peak memory consumption was still quite ok at 22G. I tried different settings to achieve lower memory consumption, at the cost of longer processing time (e.g. 8 hours and 13G peak memory).

Depending on compression settings, the final planet database was in a range of 230-265GB.

Detailed results are available on this wiki page: osm.wiki/User:Mmd/Planet_import_challenge_22

That’s all for today.

In this blog post we want to count addr:housenumber nodes across all EU member states, broken down by country. That’s a bit above 34 million nodes in total.

Queries are tested on different Overpass instances: two rather busy public ones, one idle private instance and a dev instance. With the exception of the latter, queries run on offical Overpass releases. Database contains a full planet (including metadata and history).

Query: https://overpass-turbo.eu/s/1adO

[out:csv(name, count)];
rel(2668952);rel(r);
map_to_area -> .areas;
foreach.areas->.area (
  node["addr:housenumber"](area.area);
  make data count = count(nodes),
            name = area.set(t["name"]);
  out;
);

Result: https://gist.github.com/mmd-osm/5219028c9fa93db17ac7c271a99183a1

Runtimes

kumi.systems:     gateway timeout after 1h
overpass-api.de:  gateway timeout after 2h
private server:   timeout after >2h
dev branch:       30s

NB: dev branch is exploring alternative implementation options and is currently not released.

A new version of CGImap, the rails port acceleration layer written in C++, has been released!

Full release notes: https://github.com/zerebubuth/openstreetmap-cgimap/releases/tag/v0.8.0

Emojis are only a fingertip away on your iPhone… it was only a matter of time until they show up on OSM as well. The following query finds all those emojis - worldwide: http://overpass-turbo.eu/s/l6r

No more of this…

(Image by zerebubuth)

THIS FEATURE IS NOW OFFICIALLY AVAILABLE IN OVERPASS API 0.7.55

Again a bit of prototyping today.

I always wondered what the best way to download all objects along some GPX track (more generally a list of lat/lon pairs) in Overpass API might be.

Usually, you don’t want to create some complex poly-filter for this, or even worse upload your gpx track as OSM way just for the sake of finding out what’s close by.

You might have come across the around filter, which comes in handy to find objects around a center point. I have extended this idea a bit to also handle linestrings.

Here’s a first result: a turquoise circle in the middle is our great made up GPX track, consisting of about 20 points. All the yellow stuff in the background is what Overpass API returned as highways up to 500m away from our circle. Obviously, that circle doesn’t exist in OSM, so we’re really looking at what’s close to our virtual way.

Circle

Live demo: http://overpass-turbo.eu/s/yt2

Road trip

… See full entry

(Clickable roads demo is no longer available.)

Emojis are only a fingertip away on your iPhone… it was only a matter of time until they show up on OSM as well. The following query finds all those emojis - worldwide: http://overpass-turbo.eu/s/l6r

Thanks to tyr_asd for the MapCSS enhanced version below

A bit more serious: how about searching for mixed language names like name=Vinh-Loi Asien-Supermarkt 榮利市場 : check out the following query! http://overpass-turbo.eu/s/l6s

Do you find regular expressions with full Unicode support useful and want to see them in one of the next official Overpass API releases? It’s your turn now, vote for it on Github or leave a comment here!

(demo no longer available)