After many years, I finally decided to dive back into running a full planet import using the Overpass API.

To keep things budget-friendly, I rented a modest Intel-based server for around 40€/month — nothing fancy, just dual data center SSDs and 64GB of ECC RAM.

What surprised me was the runtime: the initial import using release 0.7.58.5 took a whopping 33.5 hours — at least 10 hours longer than I had anticipated. I managed to shave that down to 26.5 hours by tweaking some settings, like enabling LZ4 compression across the board and increasing the chunk size parameter. It helped, but clearly, there’s still room for optimization.

I continued testing with my own experimental Overpass fork, that includes support for PBF, multithreading, and many other changes under the hood. Initial measurements looked quite promising with 10.5 hours total runtime. After some further analysis and improving some data structures, the import took 7 hours and 23 minutes. Peak memory consumption was still quite ok at 22G. I tried different settings to achieve lower memory consumption, at the cost of longer processing time (e.g. 8 hours and 13G peak memory).

Depending on compression settings, the final planet database was in a range of 230-265GB.

Detailed results are available on this wiki page: osm.wiki/User:Mmd/Planet_import_challenge_22

That’s all for today.