I think this is curious enough to be posted somewhere.

We're running an OpenStreetMap tileserver at osm.trail.pl with a very nice mapnik style designed by my friend. We have been rendering only a single country area until recently and decided to expand to the full planet. After a couple of weeks of occasionally wrestling with the amount of data in postgres after the planet import, we now have it running the first update from hourly diffs. It has about three weeks worth of diffs to catch up with but it's on the right track. It's still going to take three or four days.

In the meantime I decided to create a new database and reimport the geofabrik country extract that we had been using before, because people started complaining about lack of updates. The import only takes some 20 minutes. Now this database has only some 8 hours of diffs to catch up with OSM, but osm2pgsql was crawling when I launched it, and it looked like this tiny update was going to take about a week. I tried to find the query which was taking so long by SELECTing from pg_stat_activity. The culpable query seems to have been UPDATE planet_osm_ways .. WHERE nodes && ARRAY[$1] ..; I ran the query with \timing on and it showed about 10s to execute on the country extract. I ran the same query on the planet database and it took 0.4s. EXPLAIN showed that on the country extract database it was using a sequential scan instead of using the planet_osm_ways_nodes index. I learnt about disabling sequential scans with SET enable_seqscan = 0; but it kept using sequential scan anyway. When strange things happen with postgresql there's usually one person in #postgresql who can explain it, his nick is RhodiumToads. And he did explain, and the explanation is this:

… See full entry

Zapraszamy na warsztaty OpenStreetMap w ten weekend w Warszawie! Będzie to głównie wprowadzenie do mapowania od podstaw ale również weterani OSM nam się przydadzą więc zapraszamy wszystkich. Mamy już zapisaną pewną grupę uczestników ale przyjdźcie nawet jeśli zdecydujecie się dopiero dziś (możecie puścić mi informacje aby pomóc w organizacji). Jutro jest dniem terenowym i spotykamy się o 11 rano przed wejściem do Sadyba Best Mall, a niedziela wstępnie przeznaczona jest na wprowadzanie danych i teorię.

Możliwe, że będziemy też wykonywać fotografie z powietrza jutro po południu, więc szykują się ciekawe zajęcia.

Just a quick note that we're organising a two day OSM workshop to introduce people to mapping, this weekend in Warsaw, Poland. So if you're close, feel free to come whether you're a newbie or an experienced mapper. Tomorrow is the surveying day and Sunday is theory and uploading data. If you decide just now, feel free to come over, we'll be starting out of the Sadyba Best Mall entrance 11am tomorrow. You can drop me a note to help accounting.

Additionally we are likely to try kite photography tomorrow and balloon mapping on Sunday if the weather forecasts don't change. On Sunday food is provided too :)

I decided to learn about the current state of the OpenAerialMap revival, read the docs and managed to register the bit of imagery we made during the Koluszki OSM Mapping Party in Poland last week. It's not really difficult although you need a little digging to get around the current API. Most information you need to submit your imagery is found at http://docs.oam.osgeo.org/storage/creating.html (Note there's currently some inconsistency in the docs in the way the term Archived Imagery is used)

Slippymap using leaflet-js: http://openstreetmap.pl/balrog/koluszki-kap/
OpenAerialMap imagery URL: http://oam.osgeo.org/image/118400/
JOSM imagery URL: tms:http://wms.openstreetmap.pl/balrog/koluszki-kap/

Note the rectification is of very bad quality, I didn't have much time to do it all correctly, and since there's no other usable imagery of Koluszki available to us, I have only used the GPS traces collected at the mapping party as the georeference. I guess it's better than nothing although it's of much lower standards than most imagery in OpenAerialMap, but it can be a good stress test too, with the high resolution in some spots. I have much more rectified kite imagery from 2011 and 2010, but it's on my crashed harddisk which is at a data recovery place right now.

… See full entry

I don't really want to contribute to the license debates, but this is a funny analogy someone came up with on IRC, slightly coloured. :)

Imagine a village with a big, active group of people caring for a pet together, let's say it's a dog, one of those purebred dogs you see in dog show competitions, who is the village's mascot and who gets its hair cut every now and then. There's a community of villagers who maintain a fund for food and other services for the doggie and they all enjoy it and they enjoy seeing their pet grow bigger and win competitions.

Now the doggie has grown up a lot and the project has come a long way since the dog was a puppy and some people started calling for a change of the dog's haircut. They think the old look is inappropriate for a dog of this size and that it looked a little silly from the start. Some people have the opposite opinion though, but both groups are a minority, most villagers don't really care. However, it happens that the owners of the kennel are pro-new-haircut. A lot of people support them because they have taken good care of the dog's place and they became an authority in the village. A process was outlined whereby once there is enough support for the new haircut, the dog hairdresser will visit the village and give the dog a new look. The process has been taking quite long however and there isn't enough support for the change yet. Many people have accepted the change because they like the new planned haircut better, but even more accepted because they support the kennel owners and some did because the process was dragging on for so long, or because they didn't want to upset others.

… See full entry

Just a short note that a friend of mine in Ecuador told other teachers at her university about OSM and now they have used it to make the map of Salinas (see the map link) and will use it for some sort of exposition of their map project at the Salinas campus of Universidad Tecnológica Equinoccial (UTE). I don't know anything about the project but you can probably visit if you're in the Salinas area.

Just thought I'd dump them here..

* Last weekend, March 5, in Łódź, Poland we celebrated the first meeting of Polish openstreetmappers bigger than 5 people (yes, we're a little bheind but we're getting there) and at the same time we have formally incorporated the local OpenStreetMap Association which may become a OSMF local chapter at some point in the future but in the first place will help us in contacting local institutions, funding local projects, being a point of contact for local companies etc. The association will become fully legally registered in a month or two. We talked to the press, held a wanna-be mapping party and had a couple of interesting presentations including one by a commercial osm-based navigation vendor. User:tomianek has been selected as the board chairman among other reasons due to his good relations with some local institutions. I was elected the treasurer. Personally I'm a little disappointed with all the stress on promotion, formal contacts and funding, and lack of stress on actual mapping projects.

* Schuyler Erle announced on the OpenAerialMap oam-talk mailing list that he and Chris Schmidt joined MapQuest with the specific goal of working on rebooting OAM and that initial results can be expected as soon as in a couple of weeks. Yay!

* Also, Jeffrey Warren started a series of interviews with grassrootsmappers at the Grassrootsmapping.org/PublicLaboratory blogs, see http://publiclaboratory.org/report/mapper-interview-dawn-mckinney

* Also, Spanish mappers imported Corine Land Cover forests polygons and Slovakia mappers are in the process of importing CLC / Urban Atlas data.

If anyone's interested, DLT and Autodesk have a free short e-book on the basics of LIDAR. It seems quite hands-on and practical, not just dry theory:

http://www.dlt.com/library/whitepaper/lidar-for-dummies-book

… See full entry

See The Pottery Club for context

Imagine a small town with a very active club of cardboard box makers. The club members make and supply cardboard boxes for the town's people and the people are mostly happy with that. There are some other box suppliers, but their boxes are not recyclable and are actually closed, so you can't put your Garmin GPS in one of those boxes. You need an open box to do that and the Cardboard Box Club makes such boxes.

… See full entry

I posted some information about the MediaLab Chrzelice (Culture 2.0 Camp) here but now that I think of it, it was also a sort of a Mapping Party, even if very chaotic because I'm a poor organizer and because of the rush. (also most likely the first osm mapping party in Poland)

So here are some thoughts about it as a mapping party and some of my impressions, in case they're useful to anyone else making a similar event.

… See full entry

The Culture 2.0 Camp in Chrzelice, Poland finished yesterday. It was a MediaLab-style experiment in form of a four-day camp filled with workshops. Chrzelice is a (now) tiny village with a rich 700 years-long history. One of the workshops was concentrated on digitising this history and preserving the local cultural heritage. One of our aims was to digitise and archive some fragments of that heritage but in the first place it was to come up with good and sustainable tools and methods of doing that. People with a range of different backgrounds took part in that workshop: historians, sociologists, ethnologists, librarians, programmers and many more. As a result the range of tools that we used was also varied:
* recording social interviews with members of local community, some of whom remember the good days of Chrzelice castle in early 20th century (today ruined)
* scanning and photographing old documents and objects of historical significance, with engagement of a few employees of the Silesian Digital Library and the Warmian-Masurian Digital Library
* 3D scanning,
* building a map of the area in OSM.

In addition to the basic map features we put some stress on the monuments of the past and important historical objects, as well as local (even colloquial) place names that are in common use by the ~600 inhabitants, but which never appeared on an official map. It's worth noting that most of the other online map services lack 90% of the village streets in this region.

… See full entry

I posted some pictures from our first step in BAP at http://unadventure.wordpress.com/2010/08/08/bap-test-flight-1/

I hope to perfect the tools and techniques enough until Wednesday so I can produce some simple, but usable ortho imagery of Chrzelice, Poland during the Culture 2.0 Camp this week, specifically to aid the mapping that will potentially be done during the digitalisation workshop.

So I posted a little summary in English of the buildings and trees tracing and importing in Szczecin on my blog: http://unadventure.wordpress.com/2010/06/20/mostly-done-with-szczecin/

Posted it there because it's more from a personal point of view than a technical summary. You can find another summary, in Polish, but including some nice screenshots at the OSM blog at blog.openstreetmap.pl/2010/szczecin-najbardziej-kompletnym-miastem/

Now I need to catch up with all the things I noted down while surveying in my own city, and had no time to input into JOSM till now.

As documented at osm.wiki/WikiProject_Poland/Warszawa#Linie_kolejowe_i_tramwaje (Polish) I have now assigned the tags tactile_paving=yes/no and (if yes), tactile_paving:slab_size=35x35 or 40x40 (cm square) according to the data from the Warsaw tram operator that arrived in my mailbox in December, which I requested in August -- nevertheless kudos to them. The names of nearly 500 tram stops equipped with tactile paving was printed on 16 pages of paper (yes, those rectangular white sheets).

Next thing, now that I know we have all or nearly all stops in the city, is to script creation of tram route relations and keeping them up to date using the schedule data availabel through the operator's web api. Hopefully the script will be reusable for other cities and for bus / train routes too.

My current idea is to store (outside of OSM database) the fragments of routes between each two adjacent stops and combine them to form a full tram / bus line route when updating. Where there's no connection between two stops stored yet, just use shortest path in a graph to find a route, later when a human editor fixes the relation, the new route fragments will be stored on next update.

Earlier I compared dealing with planet files compressed with bzip2 and gzip, and user Mungewell suggested trying out LZO (which I had originally remembered as LZMA, which is actually a different compressor with the opposite goals: maximize compression ratio, regardless of processing time). LZO turns out even better than gzip because it compresses and more importantly decompresses even faster, again at a loss in compression ratio but without leaving the order of magnitude set by gzip, bzip2, lzma. Recent planet sizes shape as follows:

raw 150 GB
lzop 14 GB
gzip 10.5 GB
bzip2 6.5 GB

...with lzop decompressing at least 2x as fast as gzip (which is already at least 15x faster than bzip2), so on a (2009) average hard-drive and average desktop CPU, processing a planet (reading off HD + decompressing) is fastest with LZO. Compression + writing to HD is also the fastest with LZO on my hardware, unfortunately I can't give exact numbers because I'm doing my processing on one of my university's machines now, which have better specs.

I suspect with LZO we're close the sweet spot and with one of those slower HDs you might be better off using gzip because the balance between CPU speed and HD access speed is moved in the direction where you want to save on IO. You definitely don't want to use raw planet because IO becomes the bottleneck even on fastest avilable hardware.

When I acquired openstreetmap.pl I noticed lots of other ccTLDs were already taken so I thought I'd get as many as I can before some domain troll gets them because I had problems with those before. I only got .pl and .hu, most other domains were either really expensive, required a passport number of the country or something else (.hu required sending some documents in Hungarian by snail mail but I could do that). So I've set up a very simple Polish page on opesntreetmap.pl that does nothing fancy but has the initial view set on Poland and localised links, and then redirected .hu to the same page only adding javascript for changing the initial location on the page to point at Hungary.

So, if anyone has ideas for a better use for the .hu domain or can at least translate the copyright notice and the "edit" and "forums" links to hungarian then I can host a localised page on the same computer or we can figure out something else.

(posted from the Gran Canaria Desktop Summit where there was going to be a mapping party but it doesn't seem like enough people know about it and on Thursday when it was originally planned some people will already be taking off to the State Of The Map, so once again it seems like I'll never have an occasion to attend a real mapping party)

Perhaps it's an old thing but I just learnt about it today:
http://www.gpsdrawing.com/gallery.html

^ Here, plenty of (un)useful GPS traces for you.

(Posting from Gran Canaria Desktop Summit)

So I visited User:Mala in Poznań for the weekend (which we stretched to four days) and together we had our first formal Mapping Party, in the sense that more than one person had been out in the "real world" at the same time in the same place, with the specific goal of putting stuff on the map. We've been writing down everything and more, and then loading up JOSM on my laptop and mapping it before wandering off too far so if we had any doubts (my handwriting is really bad) we could just look again. The density of amenities and POIs in the centre of Poznań is such that you can map a whole lot without walking a distance longer than 100m. This also means that most stuff will not get a chance to render on mapnik because the renderer tries to avoid drawing things one on another, but this is what the centre looks like now:

osm.org/go/0ORAyZuhg==
(compare with osmarender)

In two weeks Mala is visiting me in Warsaw and hopefully we can do the same to the Warsaw Old Town market square which, like the Poznań Old Market is all pedestrian traffic only so you don't go there often except for tourism and so the seemingly busy and historically rich area is not really well mapped (osm.org/go/0Oy6Vx9yh==?layers=0B00FTF).

I've eventually set up my own live mirror of the planet file today and needed to write the update scripts etc (I could probably have downloaded somebody else's from somewhere but nobody answered quickly enough on IRC).

I first tried to have osmosis take the bzipped planet, patch it with the daily diff and generate a new bzipped planet (yes, I took care for the bzipping and unbzipping to happen in separate processes so they could run on the second core). The decompression speed was ok, the processing speed was amazingly ok too (despite java) but compression was taking 80% of the three processes' total CPU time, the ratio was about 10/15/100 (bzcat/osmosis/bzip2). It would have taken likely more that 20h to for this to complete, the result would be about 6GB.

Then I tried saving uncompressed and the thing became IO bound by my SATA disk I think, might have finished in 5h perhaps and occupied about 150GB.

Now I thought the best option would be a compromise using a really cheap algorithm, which should still be useful considering the planet is all text. I considered gzip and lzma, the benchmark here: http://tukaani.org/lzma/benchmarks made it pretty clear that lzma was even heavier on time than bzip2, and that gzip -1 (or --fast -- the lowest compression ratio setting) is clearly what I wanted. Both compression and decompression is multiple times faster than with bzip2 or lzma in --fast mode. The compression ratio is still in the same order of magnitude with bzip2 and lzma (not more than 2x worse), enough to pretty much guarantee that the whole conversion won't be IO bound on an average SATA 1TB disk and an Athlon 64 2X. The ratio of cpu cycles consumed (bzcat/osmosis/gzip) is quite sane now, 35/100/15. If the benchmark I linked is right then also reading the new planet.osm for further processing should be multiple times faster than either of: uncompressed, bzip2 or lzma.

… See full entry