OpenStreetMap logo OpenStreetMap

Zverik's Diary

Recent diary entries

Reminding you that there is a big conference coming up: the State of the Map Baltics 2020 in Riga, Latvia. How big? Well, half a thousand attendees have already registered. It’s time for you to buy a flight and meet us and other GIS enthusiasts and professionals on 6th of March:

https://2020.sotm-baltics.org/

Why so big? Well, it’s just a part of a bigger conference: Baltic GIT is not related to version control systems, but to geospatial technologies. Think of it as a partly local FOSS4G, only with less focus on open source. And it’s free!

We’ve got Allan Mustard of OSMF, Stefan Gustafsson of ESA, Tomas Straupis, Kirill Bondarenko, Danil Kirsanov, Evgen Bodunov, Rihards Olups and dozens of other speakers and OSM members. Do visit, and let’s enjoy Riga together!

OWG Must Be Destroyed

Posted by Zverik on 11 December 2019 in English. Last updated on 12 December 2019.

Note: this post goes too far and may be triggering. I believe OWG does great stuff, but is too powerfull with too few resources.

Thanks to a discussion in OSM Belarus telegram group, I suddenly realized why the new Board would not change anything. We could replace all seven members, and the project will stay the same. We could invite all GlobalLogic employees to run the project, and they wouldn’t harm a node. We are looking the wrong way.

All real work in OpenStreetMap is done by working groups. They decide on license terms, press relations, data policy, and technical stuff. Since we’ve got few volunteers, anybody can join any working group and… start working, I guess. The concept of “working” is what stops people from joining, and many people tried to explain there isn’t that much work.

In his manifesto Steve Coast suggests, besides closing off the Board and the tiles, to drastically increase budget for the Operations Working Group. Which makes sense: they are the only working group to have consistently spent money, which had measurable positive impact on the project. Why wouldn’t we want more servers?

OWG also manages our website. And planet extracts. And our data model. And developer relations. And has a final say over who can and who cannot integrate with our core systems. Which… Is a lot, don’t you think? They must have a vast experience on working in open source, on encouraging developers to join the effort.

Except they are doing exactly the opposite. Despite having 37 tile caching servers, our tiles are slower than ever, with many experienced contributors having moved to alternative tiles. In the past nine years I’ve seen dozens of developers driven off the Rails Port website code. Even active members of the Board were rejected their valuable contributions, like an endpoint to retrieve deleted objects. In their repositories, OWG members are consistently breaking every rule of open communities, by not giving outside people even a bit of control.

See full entry

Juno Leftovers

Posted by Zverik on 21 November 2019 in English.

As you might know, as of this week, Juno is no more. It is ceased to be. We had a good run, helped thousand of NYC drivers earn money and provided them with a human, attentive support team. But the increasing regulations in the city benefited Uber and disadvantaged their competitors, especially small companies like Juno. So we decided not to continue this fight.

What happens to the GPS tile layer we were making? It is still there — but not updating anymore. I have loaded two days worth of tracks for the central part, and month for the rest. It should help validate some of the complex cases around the edges, but obviously won’t tell you what has changed in recent days. Sorry for not updating it, and I hope it still helps with mapping New York.

In the meantime we have published another open-source thing: a reverse geocoder. In fact, it seems like the first ready to use, open and strictly reverse geocoder. Unlike others, it is in no way optimized for forward geocoding, but does one thing better than the others: finding an address for a coordinate. You can learn more about why it is not an easy task from my SotM lightning talk. And head to the github repository to see the queries and test cases for yourself:

https://github.com/gojuno/jrg

The moment Juno was over, the whole R&D team, me included, was acquired by Lyft. Which means, instead of asking others to open more data for OSM and to help the community with improving the map, I get to do that myself. Just for the US though, but still better than NYC only. I’m pretty excited for this new opportunity, and looking forward to working with their mapping team and their data — and making OpenStreetMap useful and better for everyone.

Илья показывает на слайд «OpenStreetMap»

На прошлой неделе меня по работе отправили в Бобруйск. Это такой большой маленький город, где делают вкусный ирис, а с недавних пор мы в Juno Lab помогаем там делать умных школьников. Рассказывал, как обычно, про OpenStreetMap. Пришлось много тренироваться, чтобы дети смогли высидеть полтора часа и не заскучать. Разумеется, стоило им открыть iD, как их школа пропала с карты — но затем появилась, заново нарисованная.

Чтобы результат не пропал, повторил весь текст на камеру. Сорок минут отборной картографии. Теперь есть что присылать людям, которые не знают про OpenStreetMap, а вам лень объяснять:

https://www.youtube.com/watch?v=469pkgFDYhk

Juno opens its GPS traces to aid in mapping New York City

Posted by Zverik on 19 March 2019 in English. Last updated on 5 July 2023.

It all started with a bad edit. We in Juno rely on routing over OpenStreetMap roads, and notice every change that breaks the map matching. One day, we encountered a pretty big breakage caused by reversing some one-way roads. Turns out, all sources available to mappers, and even proprietary alternatives, like Google Street View, were obsolete enough to validate the edit. I told the story in my FOSDEM talk, but the main point is that it gave me an idea.

How fresh a source can be? We’ve got plenty of these, but they all are very old. GPS traces come very slowly, and people still trace lines from ten years ago. Satellite imagery is updated once every few years. Street View photos (that is, Mapillary) is always older than you’d like. If something, like a street direction, has been changed in past two months, you’ve got no way to know it and reflect it on the map, save for surveying it yourself. It’s okay for a small village, but not for New York.

Overview of Juno GPS traces over New York City

See full entry

A transcript of the SotM 2018 podcast

Posted by Zverik on 4 August 2018 in English.

Victor, Ilya, Vladimir, Timofey, Daniil

“There are empty territories, but then a question arises: if they are completely empty, then why should they be mapped? For some reason, the local people decided that they don’t need to map that.”

“We, as white men, are very privileged, and we simply do not see obstacles.”

“Maybe in some form we will use the technologies made back in 2008-2009 for Osmarender”

“Why women can not program themselves an editor understandable to them, use tagging schemes that are necessary for women…“

“What this is about, is that Mapbox ruined the vector tiles for OSM.”

“To sum it up, we see that OSM is turning into Google Map Maker.”

Last week there was a big OpenStreetMap conference in Milan, where I obviously went. There were a lot of great talks, I met a hundred awesome people and shared many ideas. During the conference I wrote a live feed with thoughts and photos, though in Russian. That’s why I don’t plan to write a separate blog post: I’ve written enough.

See full entry

On Vector Tiles

Posted by Zverik on 29 July 2018 in English.

This is a very rough translation of parts of the discussion that happened on the Shtosm Telegram channel. I’ve translated it so the upcoming vector tiles BoF section does not miss my point of view, while I’m presenting OSM Streak in another room.

Kate in her keynote mentioned the recent article by Richard about what OpenStreetMap needs:

http://blog.systemed.net/post/15

The answer is vector tiles, he writes, but before that he makes a long intro about OSM not being a map; it’s a community and possibilities. He is right. But if we talk about tiles, you should understand one thing. Developers of OSM Carto style have been discussing switching to vector since April. They approach it as a technical task: to repeat the same thing, but with a different stack. And Richard’s point is that it would be wrong.

Style developers are fixated on quality, on cartography, on target audience. The make a product: tiles, which are featured on thousands of web sites. And this product, being showcased on our website, is overshadowing the real product that we make: the geospatial database. That is bad, since people believe OSM is tiles and mapping style, not data. They expect from the website to show traffic, to feature a ruler and satellite imagery. People feel that our product is the website, the routing feature, the mapping style.

What we need is to push the style to the sidelines, and to employ vector tiles. Though not to repeat what we already have in them, because then we’d get exactly the same: a product that overshadows the data. We should do intentionally raw tiles, which adapt to user’s needs. Roads, buildings, some POIs by default. But with a few clicks you could see bicycle routes: make a photo and go cycling. A map like a clay: very plain, but take it — and the possibilities are endless.

See full entry

Maps Update: April 17 → May 13

Posted by Zverik on 16 May 2018 in English.

A new minor release of MAPS.ME is coming, and just now I’ve got the new data for it — with numbers on size and features difference with the old data. First, sizes:

  • Indonesia_Central.mwm: 107 up to 127 MB
  • Tanzania.mwm: 212 up to 225 MB
  • US_Indiana_North.mwm: 19 up to 25 MB

Basically the same as the last time, except Portugal was replaced by Indiana state. And there was some prominent activity in all of Peru.

See full entry

Not Yours, OpenStreetMap

Posted by Zverik on 8 May 2018 in English. Last updated on 11 May 2018.

This is a translation of this article in Russian. Far from perfect, but understandable, I hope. Written by Ilya Zverev, CC-BY.

Riding the wave of Google Maps pricing news, Tom Chadwin reminded everyone of benefits of using open source solutions, and concluded with these words: “You now have a concrete compelling argument to those who have always asked: “Why not just use Google Maps?”.

Have I got a platinum argument towards a metaphorical Google Maps: because your open map does not have any future, that’s why. It doesn’t even have good POI coverage, unlike Google, which has franchise owners lined up with location offerings. Because it presents not a dozen grumpy dudes turning every data contributor down, but a nice mat with “Welcome” on it.

That’s an exaggeration, of course. We’ve got a great, beautiful map, which in many areas not only excels — it doesn’t have any alternatives. Nowhere else can you get a reasonably correct road graph. No other map would allow for estimating population density. Nobody would provide you with the data to install a copy of a service in a closed network.

With that, it is hard to not notice that OpenStreetMap is dying. Not because we’ve got a database for a map, or that we don’t have moderators, or the data is not split in layers, like Serge complained. For a technically skilled person it’s impossible to believe in the fall of OSM: the data is detached and decentralized, which is eternal by definition. On top of that, it is free (as in beer) and a million editors contribute to it: why isn’t every website using it?

See full entry

Maps Update: 17th of April

Posted by Zverik on 24 April 2018 in English.

Once or twice a month I build data for MAPS.ME application. After the process of downloading OSM data, building coastlines, extracting objects, generating indexes and packaging mwms is done, the final stage is testing. With that, I get notified when a file size changed dramatically, or many objects of a same type were added or removed in a region. I think you might be interested in some of these changes.

So, what happened between 16th March and 17th April?

  • Indonesia_Central: file size increased from 67 to 107 MB
  • Portugal_South: decreased from 56 down to 53 MB
  • Tanzania: 183 up to 212 MB (and it was just 87 MB a year ago!)

And in objects of specific types:

See full entry

Welcome to OpenStreetMap!

Posted by Zverik on 20 April 2018 in English.

— Yay, new mapper! Welcome to OpenStreetMap!

— I see you are using wrong tags. Please read Map Features and don’t come back until you’re done. I don’t care if it’s too long. Go to Google Map Maker if you’re intimidated.

— I don’t know you. Please go to talk@ mailing list and introduce yourself and reasons for your mapping. Yes, I know you have to sign up using some obscure web page, and then receive unwanted mail for eternity. But rules are rules. Go do it, or leave.

— I see you work for a mapping company. I will be watching you, because you do not care for our map. None of you do. Yes, I am working at such company too, but I care.

— I caught you not tracing from Bing like the rest of us. Please explain your sources and acquire specific permissions for using these. Or better stop.

— I see that among 500 buildings you’ve traced, 20 have been recently demolished. I know that they are still on imagery and in the registry. But it looks like your quality of mapping will never reach mine. Please go away and never come back.

— Hey, I’ve seen more edits from you. I think we’ve discussed it. I don’t care if they are good, I have reverted them. Please do not come back.

— Hi new mapper, welcome to OpenStreetMap!

Turn restrictions are pretty common to OSM, them being the second most popular relation type, after multipolygons. Usually a restriction is a relation with two highway ways, “from” and “to” and a “via” member connecting these. The value of “restriction” tag adds a meaning: which kind of turn is forbidden on this route. For example, “restriction=no_left_turn” (and any other no_* value) forbids going from “from” to “to” way using “via” intermediate objects. Alternatively there are “only_*” values, forbidding any routes from the “from” way except the one leading to “to”.

Usually “via” members are nodes, which makes them redundant for all types of restrictions except “no_u_turn” (which has the same way in “from” and “to” roles). Thus supporting them in a routing libraries has been easy. But — a “via” member can also be a way. For example, on a dual-carriageway, like the one pictured below. Supporting ways for “via” members is hard, but they constitute less than 3% of all restriction relations, so developers have often ignored these.

See full entry

The Subway Validator evolves

Posted by Zverik on 19 March 2018 in English.

First, thank you all for improving the mapping of subway networks all around the world! In just half a year, we made more than 150 networks routable, of 180 total. That is very impressive. Today, OpenStreetMap has more data on subways than any other source, open or proprietary.

Last week, I have made a few improvements to the validator. The major one is a change to how stations are counted. We had an issue of transfer stations: for some cities they were counted once, for other — twice, depending on how they are mapped. This simplified calculating a projected total number of stations (just copy it from wikipedia), but affected mapping.

Now, thanks to disc12, it sums up numbers of stations for each line. This is more predictable and allows for different interchange mapping styles. In the spreadsheet, counts of stations have been mostly updated in form of a formula: =line1+line2+…+lineN. You can clearly see how many stations it expects for each subway line. If you find an error there, add a comment and I’ll update the number.

Having trouble with missing or extra stations? Click on “Y” near the city name, and you’ll get a YAML file with all stations, transfers and lines. What’s new is a number of stations for each line (calculated as a number of unique stations for all its itineraries), along with a list of stations. Comparing it to wikipedia is much easier.

There are some improvements planned still. For example, handling of stations under construction: you cannot add these to routes at the moment, or you’ll get an error. And there is a “nowhere near the tracks” error that is hard to track — I really should do something with it. And the preprocessor calls for a GTFS output.

Thanks for mapping, now let’s finish the last cities and then monitor the world for new subway and light rail stations. If you are an app developer, please consider using the validator output for your app. Contact me if you have any questions.

Subway Routing in Maps.Me

Posted by Zverik on 25 December 2017 in English.

Last week we published the latest version of Maps.Me. It’s got the major version number increase — 8.0. And not for nothing: there is a brand new “Discovery” button, which shows interesting places around you. There are christmas markets on the map — not from OSM though. You can register as a “local guide”, meet new people and show them around your city. Hotels from booking.com can be filtered by price, rating and availability.

But that is not why the release is worth celebrating. The main thing is, we’ve made a metro routing!

Screenshot of an Underground route in London

It won’t be an exaggeration to say that this is the first even public transport routing application that uses solely OpenStreetMap data. Anybody can employ GTFS data, but using OSM is not that easy. All these relations — “route”, “route_master”, “stop_area”, with enourmous tables in the wiki detailing their usage. Utter mess in the data, a result of mapping for a renderer. Very few people understand public transport mapping, so how did we even use it?

See full entry

I broke my streak

Posted by Zverik on 19 December 2017 in English.

A few days ago I forgot to spend a minute in an OSM editor. I had almost a month of consecutive edits, level 3 in OSM Streak with almost a hundred points. And then, after a hard day, I forgot about even opening a laptop. Neither an email, nor a telegram notification helped.

Participating three days in a row

The next day I submitted my first changeset in a row. How much did I lose in the game? I had been receiving five points for each changeset (I was too lazy to complete tasks), now I get three, completing the tasks. The fourth level was two months ahead, now… I don’t know, two and a half? 500 points is a long way to go.

See full entry

A year of edits with OSM Streak

Posted by Zverik on 3 December 2017 in English.

Hi, I am Ilya and I have been uploading changesets 15 days in a row. Not because I’m so into it or somebody makes me: I’ve made a tool that reminds me to do it. In a year I expect my HDYC activity chart to be completely filled. And you can have the same too.

Introducing OSM Streak: a website that gives you points for submitting changesets each day. You get 1 point for the first changeset, and then you get more: for example, I will receive 4 points for my next changeset tomorrow. And that is not all: it gives you a random task each day, so you don’t stare at the map trying to come up with an idea. For completing a task, you get an extra point. And when you map many days in a row, you gain levels, which open more tasks.

Forgetting to visit a website is expectable, so OSM Streak is also a Telegram bot (find the link on the “Connect” page). With the bot, you can forget about the website: it accepts changesets and sends you tasks every day. Alternatively, you can subscribe to e-mail notifications, which will be sent on 1:00 UTC.

All the tasks and the website and the bot can (and should!) be translated into your language. We have English and Russian, and I would be very grateful for more translations.

Finally, there are 26 tasks at the moment, and you can add more by making a pull request or an issue on github. You can also use the source code as an example for writing a telegram bot, doing migrations with peewee ORM, sending e-mails with python or adding i18n with transifex.

Hi. I’d like to ask you to the Metro Mapping proposal page, read it in full and leave your vote below. The proposal mostly summarises subway and light rail mapping practices and introduces a few useful changes, listed in “What This Affects” section.

You will find many opposing votes there. You might be tempted to read only these comments and skip studying the material. It is simpler, and some of the authors are well-known in the community: they must know what they are talking about, right? And there are 8 screens of text, who would read that?

That is what I don’t like in the current proposal system. It does not matter what the proposal actually contains, how deep an author knows the topic or if the features in questions are already mapped in a suggested way. What a great power you have when the voting starts — strike out weeks of work by writing a few words! You must be a fool not to excercise that power. And who knows what might happen if the proposal is approved? Everybody understands you cannot edit a wiki page after that. You would need to write your own proposal, and it might be rejected, so it is safer to just oppose.

For the first time in OpenStreetMap history I started actually using the data to route users through subways. And found out the tagging is insufficient: there was no sane way to map interchanges, and you could put thousands of members into some relations. For a mapper things also looked bleak: to map a subway properly, you would have to know how underground tracks go, and where underground platforms are located. No wonder that a beta version of a validator showed that the world had like three subway systems mapped good enough, out of nearly two hundred.

See full entry

We all used building:levels and alt_name without giving it a second thought. Why are these keys built that way? Why not levels:building? To me, it looks like there is a rule for building composite keys.

Suffixes

ref is the basic tag for storing a reference number. For a reference number in some third-party table, we add a suffix: ref:third_party. That is because the new tag still contains a reference number. We have all such numbers in ref* keys. The rule of thumb is, the meaning of a value is defined by the basic tag before the suffix. ref:third_party is still a ref, and source:maxspeed is a source.

Sometimes we cannot use suffixes for historical reason. That is the case with name: we use name:en and other suffixes for names in other languages. For that reason, we build composite keys by prepending a content with an underscore: local_name or place_name. These are still names — a reversed order from the semicolon notation.

Of course, an underscore is also used for multi-word keys: public_transport and admin_level.

Namespaces

Then, there are namespaces. The most known is addr: with addr:housenumber and so on. Without a suffix, addr key has no meaning. The same with contact: and turn:. Namespaces are used for marking a group of tags that have the same meaning, have similar value formats and they are usually described on a single wiki page.

Some namespaces are used for tying properties to a part of the object described by the main tag, and for adding more specific properties of it. For example, building:* tags describe attributes of a building, and we also have roof:type and fire_hydrant:type. These words are most often put on the same object as a key or a value, e.g. building=yes or amenity=fire_hydrant, but also can mean a part of a structure denoted by these tags, like how buildings almost always have a roof.

See full entry

OpenStreetMap Awards 2017

Posted by Zverik on 28 July 2017 in English.

Did you know that the community voting for the OpenStreetMap Awards is open right now? We have chosen many nominees that did a great job last year, 45 of them! Not as many as the total number of active community members, but still a lot. And nine will receive the award at the Stat of the Map in August.

All nominees will be features in the series of OSM Blog posts: https://blog.openstreetmap.org/tag/awards2017/

Help translate the website and project descriptions to your language: https://www.transifex.com/openstreetmap/osm-awards/dashboard/

The voting closes on 16th of August. Vote now — you can change your choice at any time. Nominees would be glad to hear they are supported by hundreds of voters, so we would need at least a thousand people to vote.

Promote the Awards in your local communities, and if you are nominated, do encourage your subscribers to vote for you :) That is okay, provided you don’t pay for votes or anything like that.

Better Walking Papers

Posted by Zverik on 5 July 2017 in English.

Walking papers from the Tula Mapping Party

I have talked publicly about improvements to walking papers since at least SotM 2013. Made a blog post here in 2014 with some thoughts. But all I’ve seen were new ways to print tiles or atlases. While I admire the Field Papers and MapOSMatic fork improvements over the past years, a good walking paper is more than that.

For a long time I have been using a 28-step process to prepare walking papers for my mapping parties. It involved using Maperitive, Inkscape and some proprietary software. This year I finally got fed up with reanimating that old renderer, which doesn’t work perfectly on Linux, and tried something else. I had always been recommending QGIS for printing maps, and I decided to try it myself. Turned out, making walking papers with it is really simple and straightforward, albeit not without issues.

See full entry