OpenStreetMap
I’ve recently added a new feature to https://map.atownsend.org.uk , prompted by the dreaded lurgi that is affecting everyone at the moment - the ability to show pubs with either outdoor seating or a beer garden.
Outside seating is shown like this:
with a black line at the bottom right of the icon.
Here’s a beer garden:
I’ve added a couple of short but interesting scripts to github. They fetch and sort posts from an OSM mailing list from some time in the past. For example, here are the topics on the talk list at the height of the licence change furore:
listtopics.sh talk 2012 May
35 OSM : It's a shame !!!
24 Edit review: ele=0
22 Worst of OSM
21 TomTom is thumping us
17 Import of buildings in Chicago
17 Edit review: intermittent waters
16 OSM cycle map - ?excessive focus on long-distance routes
15 Edit review: "building"="levels=N"
14 Cycleways and Access tags: Left, Right, Forward, Backward?
10 OpenSeaMap
10 Cycle lanes & cycle tracks - my findings and a proposal
7 Changing capitalization (Lima)
6 OWL down
6 OSM data density - top regions
6 Old versions of OSM?
6 Copy-and-paste remapping
5 TTTracklog and TomTom core 9.4
4 OSMCoastline / OpenStreetMapData.com
4 Mapnik rendering issue with *_link roads
4 (dis)Honesty and Copyright
and here’s the tagging list today:
I’ve added a new page to the switch2osm guide: “Manually building a tile server (20.04 LTS)”. At the same time, the old “with packages” instructions has been retired from the list as that depended on Ubuntu 14.04, which has now dropped into Extended Maintenance.
Ubuntu 20.04 (“Focal Fossa”) has now been released. There are actually very few changes from the 18.04 version - mostly just updated versions of software (including postgres 12). Following these instructions shouldn’t take more than a couple of hours for small areas - the longest period of the setup is waiting for the shapefiles used by the style to download.
As before, the page is designed to be “the least you need to do” to get a rendering server working to create tiles like the ones that you see at openstreetmap.org. If it still sounds a bit complicated, don’t forget that you can try things out with Docker very quickly - and that should work anywhere that Docker is supported.
For some time I’ve been experimenting with how other information can be added to established icons. For example, here’s how I’ve been representing various tourist information items:
In those examples there are only two pieces of information - that it’s something to do with tourist information, and whether it’s a board, guidepost, plaque, etc.
With pubs, I already show several sorts of information together. For example this one:
As you may be aware, the “switch2osm” site recently moved from Wordpress to Github. This means that it’s easy to request changes to the site over at the github repository. Any changes made there will automatically go live on the main site after a minute or so.
“Requesting changes” might mean commenting on existing issues or creating new ones (which will need someone else to make the code or text changes you’ve requested), or it might mean suggesting the changes yourself via a “pull request” - a list of specific changes you’d like to see made. Here is an example of a pull request - you can see what it’s a change from and to, and on this page you can see any comments and when it was merged in and by whom.
Obviously, if people can contribute via pull requests rather than just creating issues it’s better for everyone - it means that you’re not relying on someone else reinterpreting the changes that you’d like making, and it’s quicker for the maintainers too.
Normally if you wanted to make a major change to something on this site you’d “fork” a copy of the repository into your own github area, create a “branch” from that that relates to the change that you’d like to make, “clone” that locally into an environment on a local machine that allows you to test the site locally, make your changes, test your changes in a local web browser, submit your changes back to your copy of the site, and then create a pull request explaining the change you’d like to make.
What you may not know is that for simple textual changes you can actually do the whole process online. That’s not appropriate for anything that might break links, but for simple text changes (like the example above) it’ll work just fine, and this is what the rest of the diary entry describes.
Imagine I’ve just installed OsmAnd on an Android phone and want to change the rendering slightly.
For completeness, the OsmAnd version is 3.5.5, I downloaded it from Google Play, where it’s called “OsmAnd - Offline Travel Maps & Navigation”. The Android phone is a Blackberry DTEK50 (actually a badged Alcatel design) running Android 6.
I ran OsmAnd and skipped the initial download (because I’ve already got some .obf files to install), but you’ll probably want to download some local data.
I want to copy files to and from the device, so I’ll plug it into a PC. How you copy files depends on the OS on your PC. Next, plug the device in via USB and select “copy files via MTP” from the prompt that appears on the device (this actually varies by Android version, and what options are available to copy files to and from a device depend on that and the phone itself). A window showing an MTP connection to the device contents will probably appear - check you see something in there. If you don’t, unplug it and plug it in again.
I’m running Ubuntu Linux on the PC and want to copy files to and fro from the command line, so I’ll install something to allow me to do that, check that I can read something from the device, and unmount it again:
cd
mkdir mtp
sudo apt install jmtpfs
jmtpfs ./mtp
ls ./mtp
fusermount -u ./mtp
For me the “ls” above shows ‘Internal storage’. Before copying anything used by OsmAnd in the steps below I’ll make sure that it isn’t running on the phone. How to do this varies by phone - on mine it’s “menu” and then “close”.
Next, we’ll look at what maps are currently installed. How to do this varies by Android version, but this is how things appear on this Android 6 device:
Following recent discussions on OSM mailing lists about tag homogenisation it struck me that there probably wasn’t a good summary anywhere of the tools that people use to change the tags in OSM data into something that they can use. You might wonder why on earth we need to do this, given that OSM has natural language names for everything, but unfortunately many words used in OSM might have different meanings around the world, such as “city” and “highway”. However, this isn’t a diary entry about that - it’s a brief summary of what I’m aware of and have used for converting the “super detailed” tagging in OSM into something more appropriate for e.g. something rendering OSM data.
The documentation for this is here. I wrote a diary entry about how I use this method to change the “name” language of Welsh-speaking places in Wales to Welsh, and Scots Gaelic-speaking places in Scotland to Scots Gaelic.
There are a lot of peaks mapped in OpenStreetMap, ranging from proper mountains to mere pimples. They’re usually rendered the same regardless of how high they are:
However, with a bit of preprocessing and a bit of help it’s possible to render higher ones at a higher zoom level:
I was prompted to write this because of a few discussions over the last couple of months - people talking about the default tags on various OSM elements, reactions to Amazon mappers’ additions of “last mile” service roads and discussions about how to tag England and Wales’ public rights of way network.
This is very much a personal view and one that’s very “England and Wales” centric - there are lots of different approaches to rural tagging in general and footpath tagging in particular, and many places around the world (enlightened places like Scotland and Scandinavia) have different default access rights to the curious set we have ended up with in England and Wales.
Let’s start with an example - what’s the reasoning for the tags on this way in OSM?
(following on from this diary entry)
Supposing you would like to overlay a set of boundaries over the top of any existing map style - what would you need to do?
Using this as a reference, create a database to hold the new boundary data.
If a historical data file is what you want, download the data for the desired date and rename it to a file name that the “update_boundaries” script will recognise:
-rw-rw-r-- 1 renderaccount renderaccount 459723464 Nov 2 01:46 ukraine_2018-11-01T21:14:02Z.osm.pbf
You can also make other changes to this data if you wish using tools like osmium, osmfilter, osmosis, etc.
Modify this script to process your data.
Run the script to load the data that you have already downloaded:
sudo update_boundaries.sh current
Create a version of “mapnik.xml” (or whatever style file you are using) that looks in the database that you’ve loaded your boundary data into. In this case I just changed “dbname” references from “gis” to “gis3”.
Ensure that “/usr/local/etc/renderd.conf” knows about the new tile layer. The default one I use just has “ajt” in it; I created a copy of the last section, called it “ajt3” and changed URI and XML values.
[ajt3]
URI=/hot3/
TILEDIR=/var/lib/mod_tile
XML=/home/renderaccount/src/openstreetmap-carto/mapnik3.xml
HOST=localhost
TILESIZE=256
MAXZOOM=20
Restart renderd and apache2, and your new tiles will appear. Then in a simple Leaflet map add an overlay layer for your new tiles. This is an example that shows Leaflet overlays but there are many, many others, including on the main Leaflet site itself.
Here’s what it looks like:
When I developed the map style that you can see here from OSM-carto one thing that I didn’t carry forward was the representation of borders. Partly this was because it’s not easy to separate maritime ones from non-maritime ones, the the purple maritime country borders frankly don’t look very nice:
Partly it was also due to “clutter” - I didn’t want an arbitary boundary to get in the way of a river or stream or other feature.
Sometimes however, it would be useful to see what borders are where, so why not do it as an overlay? This turned out to be rather easier than anticipated.
I’ve been wanting to write an explainer for this for a while (even before this diary comment). The problem was that I wasn’t aware of a Docker example that (a) was reasonably up to date and (b) was designed around tile serving (rather than style design for example).
However, this docker example was recently mentioned by its author on IRC. It’s based on the building a tile server for 18.04 instructions and also handles importing data and running the tile server afterwards.
First things first, I installed Centos 7 on a virtual machine. I got a “minimal” iso image for that from here. I installed it (which involved turning the network on inside the installer and selecting the disk to install on). I also set a root password and added a non-root user (I’ll use the example “renderaccount” below), and noted the IP address that it obtained. I did a “yum update” to install any available software updates.
There are a few sets of instructions for “installing Docker on Centos” - DigitalOcean have one, and there’s one here which also contains lots of other useful information. Following the instructions there:
su -
yum install yum-utils device-mapper-persistent-data lvm2 wget
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install docker-ce
systemctl start docker
If you then do this:
systemctl status docker
In the output you should see “active (running)”. Do this:
systemctl enable docker
in order to have docker start every time and then exit from root and do this from you’re non-root user:
sudo usermod -aG docker $USER
So that you can control docker from that non-root user. Log out and back in and do this:
docker container run hello-world
It should say “Hello from Docker!” to show that everything is working OK. If instead you see “permission denied” it probably means you haven’t logged out and back in again.
I’m in the process of changing map.atownsend.org.uk so that zoom level 13 can still be used as a “route planning” layer but walls, hedges and ditches don’t get in the way. This wasn’t originally a problem, but people have since mapped lots of fences, hedges,walls and ditches, and it can be difficult to see footpaths at that zoom level now. Here’s a screen showing half new tiles, half old ones:
That corresponds to here on the site and here in OSM. The new version will probably be rolled out in a week or so’s time. Other zoom levels will be unchanged - you’ll still see hedges and ditches a couple of zoom levels earlier than with OSM’s “standard” style, and paths will still be legible 2 or 3 zoom levels earlier.
I’ve added a new page to the switch2osm guide: “Manually building a tile server (18.04 LTS)”.
Although Ubuntu 18.04 (“Bionic Beaver”) isn’t released yet, it’s due out fairly soon and daily builds can be downloaded from here. There are actually very few changes from the 16.04 version - mostly just updated versions of software (including some Mapnik fixes). Following these instructions shouldn’t take more than a couple of hours for small areas - the longest period of the setup is waiting for the shapefiles used by the style to download.
As before, the page is designed to be “the least you need to do” to get a rendering server working. As before I also wrote a wiki page which goes into a bit more detail, including other things that you might want to do.
It’s also worth mentioning than there are many more resources available now than there were a couple of years ago - see for example Ircama’s tutorials, and also this guide that covers the setup of an OSM Carto renderer within Docker.
(written in response to the discussions here and elsewhere, prompted by this blog post, which was in itself presumably prompted by this HN item)
Yesterday was a nice day, so I decided to walk from Derby to Alfreton (roughly here). The criteria were pretty simple:
Ignoring “automatic routing” altogether, what map provider in the UK has that data? OSM does (at least in the area that I’m interested in). Google sort-of does - I’m sticking to roads here, and via their StreetView I can see that e.g. Codnor Denby Lane does have somewhere to safely walk, but it’s not really practical to “virtually walk” an entire route in Google’s StreetView to check it before setting off. HERE has some aerial imagery but not with enough detail. The UK’s Ordnance Survey surely has most of the data, but getting at it is hard (even if you pay them money). The small map on this page can be zoomed in to hint at sidewalk info, but neither the buyable OS Explorer map (which I have a copy of) nor this page which I presume is based on OS OpenData, do. Bing can show basic road info and their own or OS Explorer’s map style, but no “is this walkable” information.
With regard to POI data, Google was (until yesterday) the only one that had the correct name for this pub, and does link through to food and beer details (even CAMRA, usually the go-to guide for this sort of thing, are out of date). However, there’s no differentiation on the map beyond a “place that serves food” icon and a mousover blurb (“family friendly chain pub and restaurant”).
So am I missing something - is there anyone out there with this level of data that’s available in an accessible format, other than OSM?
There’s a discussion over on talk about contributions to OSM’s Standard map style. This diary entry explains what I did in order for a change to fix a problem with bus guideways that was visible in that style. It’s provided here so that hopefully it can help more people that want to (but don’t necessarily know how to) to contribute.
The first thing I did was log an issue for the problem that needed to be fixed. As well as describing the problem that also outlined the fix (actually I referenced some code in a different map style where I’d already fixed the problem). The initial discussion on the issue was helpful in that other people said that yes, they thought that it was a problem too.
However, describing the problem is only part of the way towards actually fixing it. In order to do that, I started by reading the “contributing” guidelines and the links from there.
I already had a github account. I forked OSM Carto into my area to create this. Within there, I created a new branch “guideways_low_zoom” (since deleted) just by typing the name in into the github UI.
I already had a virtual machine running Ubuntu 16.04.3 LTS (server) with all the pre-requisites in place. I’d set it up as per the switch2osm guide, but mainly use it for developing a different map style based on an older version of OSM Carto. Another option is to follow the recommendation in the OSM Carto repository and use Docker.
I then had to “git clone” my forked repository to set it up on that machine, and I then did:
git fetch --all
git checkout --track origin/guideways_low_zoom
to get the correct branch there.
In order to actually get the vanilla copy of OSM Carto working, I had to follow the instructions in the “Shapefile download” section of the switch2osm instructions (essentially running “scripts/get-shapefiles.py”).
I’ve finally got a reasonable rendering of linear barriers working - like the cattle grid in this picture:
That’s here in OSM and you can see a zoomable map here.
The key to getting it working was to ensure that the width of lines used at high zooms mostly matched the real width of the feature on the map (see here for cattle grids), except for gates and stiles which are designed to stand out slightly. Walls and fences are slightly different, allowing you to tell them apart.
It’s been possible to run Linux software on Windows for some time using Microsoft’s “Windows Subsystem for Linux”. Here’s what needs to be done to run a simple tile server there, based on the existing instructions here .
Firstly, install “Bash on Ubuntu on Windows”. There’s are some intructions here (but don’t worry where that says “This won’t work with server software”). Continue down to where it says “You’ll be asked to create a user account and password for use in the Bash environment”. It’s easiest if that user account is the one that you use in the rest of the switch2osm instructions.
Once installed, at a shell prompt “lsb_release -a” should say:
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.3 LTS
Release: 16.04
Codename: xenial
What you get is surprisingly similar to a standard Ubuntu server machine. Some things don’t work (e.g. “screen”), and you’ll have to manually start some of the services that will need, but otherwise things are very familiar.
Run “sudo apt update” and “sudo apt upgrade” to upgrade to the latest version of everything.
Next, go over to the switch2osm instructions and follow those as normal, except for the caveats pointed out below:
If you want to ssh in (you probably will) you’ll need to “sudo service ssh start”.
You’ll likely want to allow access to ports 22 (for ssh) and 80 (for http) and possibly others at the Windows Firewall (inbound rules / new rule / port / tcp port 80 / allow / all / “name”).
After installing postgresql but before doing e.g. “createuser renderaccount” you’ll need to start postgresql manually. “sudo service postgresql status” will say “down”, but after “sudo service postgresql start” it will say “Starting PostgreSQL 9.5 database server” and “service postgresql status” will then say “9.5/main (port 5432): online”.
A previous diary entry explains how I created a basic map legend for a map that I maintain that highlights various things common near me that tend not to get shown very well elsewhere - public footpaths, hiking trails, offices, that sort of thing. It struck me that a lot more things had names than almost any other map (other than OSM Bright) actually shows.
I therefore went through a process of looking to see what tags people used names with (it’s almost everything - here’s a named cattle grid), and what they used as the “naming tag” for various things (mostly “name”, obviously). The result is here:
although to get a better idea of things click here and zoom around.
You might want to set up a rendering server for a short period of time, or you might want a server that has lots of memory while you’re importing data, but much less afterwards. An “always on” Azure server will probably cost more that a comparable VPS somewhere else, but if you only want it for an hour or two at a time, it may cost considerably less.
This page describes using Microsoft Azure to do this, but of course other cloud providers are available, and depending on what you want to do may be better suited to your needs.
I have no commercial relationship with Microsoft (beyond occasionally buying hardware with Windows preinstalled and then later removing it) and this isn’t a “recommendation”, beyond providing another option to setting up a server yourself.
If you haven’t already got one that you want to use, create a Microsoft account. Jump through whatever validation hoops you have to (for me that was an email address or a phone number, but this may vary depending on where you are).
Next, go to portal.azure.com and sign in. There will probably be a few “free trial” buttons around; click one of them to get a 30-day trial of up to £150 / $200. You’ll need to provide the Microsoft account details (from above), a phone number (again) and a credit card. If you’re in Germany and want to keep your data there see here.
This page is a good starting point. You’ll want to install the Azure CLI because it’s a lot less fiddly than than the alternatives, though there is a “Cloud Shell” option in the portal (actually that’s an account on a remote Ubuntu 16.04.2 LTS VM!) and a “clicky pointy” interface in there too.
First, create a “resource group” to tie everything associated with this machine together, and make it easier to tidy up later. I created this in “uk south”, you can pick somewhere closed to you:
az group create -n MyResourceGroup -l uksouth