I just finished a two-week effort of cleaning up braided streets in Austin, TX. About 200 of them. Wow.

OSMXAPI is dead or useless recently, apparently a victim of popularity overload.

I needed to grab a whole city in the U.S. to check for braided streets. I've been using OSMXAPI in the past because the main API only allows small areas. But recently OSMXAPI has been overloaded or unresponsive. Even worse, when I have gotten through, it has been giving me error messages that a new restriction mechanism is in place. It has rejected my requests, even when they are ridiculously small (one was only 250kb through the main API).

The osmxapi error message suggested using the planet file instead. This sounded bad, because I did not want to download 4GB just to get a few megabytes of city data. But I found a great set of extract files for the US at

http://downloads.cloudmade.com/north_america/united_states

It looks like they update them weekly. Thanks guys!

I am using Osmosis to pull out my desired city by bounding box. This looks like a good solution.

As Frederick Ramm put it recently, "Look at our current map at zoom level 2 and try not to run away crying." I think I'm finally making progress.

I've built some shell script hacks to work my way through a long list of z8 tiles. It is going to take weeks at my current rate, but that's OK.

I take a swath of z8 tiles, hundreds at a time, and rebuild them. Then I rebuild the z4s they inhabit, and then finally z0. This takes several days per cycle.

I also built some scripts to find and clean "unknown tile" artifacts. I'm better, or at least more comfortable, writing shell scripts than perl, so my scripts tend to be slow and kludgy, but I think it is working.

The "unknown type" tile has a certain color in the text, so I do a quick check for it at z8 using ppmhist. If I find it, then I loop through the z12 tiles inside to see if any match the unknown type tile. If I find one, I replace the image and then use the API to flag it for re-rendering. Replacing the image is a short-term fix but it is making the map at least look a little better, a little sooner. I'm checking both the "tile" layer and the "captionless" layer. If one is unknown and the other is not, I just copy one to the other. If both are unknown, I copy in the mapnik tile. In any case, the image will get replaced by a proper rendered tile soon.

I think I can look at z2 without crying so much now :-)

I got a new GPS receiver, the iBlue 747 (apparently also known as a BT747). It is a small receiver and logger with no display. It can record to memory, and also has both USB and Bluetooth interfaces.

It is much more accurate than my previous GPS, the Magellan GPS Companion on a Palm V. The old one takes several minutes to get a signal lock (up to 5 or 10 minutes when moving), can't receive a signal inside a house or most buildings, and has large variations (like 10 to 20 meters) in repeated readings of the same location. In the car, it needed to be on the dashboard with the best possible view of the sky to get any decent readings.

The 747 is much quicker and more accurate. It always has a lock within 30 to 45 seconds, even when moving. It has good reception indoors, and is much more consistent from one reading to the next. In the car, it gets just as good readings as the old GPS when it is in my pants pocket, and on the dashboard it is very accurate. It looks like it is a little better in the downtown "urban canyons" too.

I've been using it mostly as a stand-alone logger, and downloading the tracks by USB. I definitely missed the sattelite display at first, to know if I was getting good reception. I quickly came to find that I didn't need it with the 747 like I needed it on the Magellan; the 747 "just works" without lots of adjustment and positioning.

I have tried a few Java apps on my S-E Z525 bluetooth cellphone, but I haven't liked any yet. The biggest problem is the phone has a short battery life running Java and bluetooth, and I can't stop the screen dimmer. GpsMid looks nice, but it doesn't recognize my GPS over the bluetooth. TrekBuddy works fine, but I find that with the 747's built-in logging, I just don't need an interface much.

… See full entry

I've learned a lot in the last week. I've become interested (obsessed?) with cleaning up the lowzoom levels. It appears that the addition of the captionless layer is a relatively recent change, and therefore most of it is missing. The main "tile" layer has been around a while, but as tiles in the lowzoom layers are rebuilt, they are pulling in a lot of garbage from the missing areas in the captionless layer. I understand that this will fix itself over time, but at the way things look, that could be years. It needs a boost.

The primary problem with the lowzoom builds is that when a tile is missing, the server returns a tile with "unknown type" on it. This is a good thing for the server to do when it has no other information, but in this extended transition period it makes for ugly maps.

The component tiles are in the z12 build request queue, but there are probably hundreds of thousands of them in there (just guessing). In the mean time, almost anything is better than the unknown tiles. I have started copying some land tiles from the main tile layer into the captionless layer, to fill in as a stop-gap. Yes, they will have little captions scrunched down, but it really is better than nothing. As long as the tiles are still in the request queue, they will get properly fixed later. I think it makes a good interim solution.

The problem is that there are sooooo many tiles. The total possible tiles in 17 zoom levels is 68 billion. Most of those are sea, which won't need anything smaller than z12 and rarely below z8. Even without that, there are about 3 million land tiles at z12, and I think about 10000 at z8. That's a lot of tiles to update. Fortunately most of Greenland and Siberia don't need anything more than blank land.

… See full entry

I've started running Tiles At Home on my Centos 5 Linux machine. The instructions on the wiki are pretty much up-to-date and correct.

I was a little disappointed to find out that I'm not the bottleneck. Or rather, that there are plenty of TAH clients, and the bottleneck is in the server processing. Most of my renders in the last few days have had to wait 5 or 10 minutes to upload.

So I figured I should try something more challenging, like the lowzoom rendering. This is the widest view of the areas, between zoom level 1 and 11. I'm concentrating on 8 through 11 right now. I did some digging around the mailing list archives, and it took me a little time to realize that the wiki instructions were actually up-to-date, and that I do need to run both tilesGen.pl and lowzoom_composite.pl for the whole process. Now that I have that figured out, I'm updating some lowzoom areas in my region. There are some blank blocks along the Oregon coast that I am fixing now. The only frustrating part is waiting on the server for uploading and processing. Patience, young Skywalker.

The other interesting thing I found is some big chunks of empty "land" in the middle of the water. You can see this in the Strait of Georgia, the body of water between Vancouver Island and the Canadian mainland. It appears that these are artifacts of the blank tile handling optimization, where the tile type at zoom level 12 did not get set correctly. I'll guess it is because it is inside the country boundary and so did not get treated as ocean. I found that I can construct 69-byte sea tile files and upload them. I just got my first results out of the server and it is working to fix it. It should not be too hard to clean that up.

I just starting doing Tiles At Home. I have a Linux box running Centos 5 with 1 GB of ram, so I figured it would be worth doing.

I wonder if there is a handy way for me to give priority to rendering my own edits first? Hmmm...

I have finished correcting the "braided" streets listed on the Tiger fixup page.

I've been looking for more to fix, and I haven't found many. I found a handful more in the greater Bay Area, mosly in Oakland. None in Portland. A few duplicated streets in Tacoma, but no actual braided ones anywhere around Tacoma and Seattle.

I'm going to broaden my search across the US, but I'm wondering if the underlying issue was peculiar to the San Francisco area. Since the TIGER data is an amalgam of other sources, it could be that the topology leading to the braiding phenomenon only occurred in some source data from that area. Interesting.

I have written a perl script which can fix a "braided" street in US TIGER data, as described at

osm.wiki/index.php/TIGER_fixup

It is a command-line tool which will download and fix the ways, and produce a file which you can open in JOSM to verify and upload.

You can find it at

http://svn.openstreetmap.org/applications/utils/filter/osm-unbraid/osm-unbraid.pl

Please give it a try and let me know how it works out.

I am working on a perl script which can fix a "braided street" in US TIGER data, as described at

osm.wiki/index.php/TIGER_fixup

It is a command-line tool which will download and fix the ways and produce a file which you can open in JOSM to verify and upload.

It should be ready for others to try out within the next few days.

Ideally it would be a JOSM plugin, but I don't know Java. Yet. This just may be the project that pushes me to learn it.

I found this at

http://forums.macrumors.com/showthread.php?t=142985

I just discovered that the Mac Mighty-Mouse can be configured to do the middle click in JOSM. JOSM needs the middle click to show way numbers. I can't find any other way to discover a way number in JOSM, but it is free software so I won't complain.

In the Mac OSX System Preferences keyboard & mouse panel, the Mighty Mouse scroll-ball can be changed to "Button 3". For me, this worked to get JOSM to display the middle-click info box.

I took a little outing at lunch time with my GPS and rode the south end of the Portland Streetcar line, and the new aerial tram up to Pill Hill. Now they are in the map.

Fixed the Portland east side Max lightrail tracks from the Steel Bridge to the Gateway Transit Center. I unraveled the interchanges at each end, and fixed the spots where the train tracks were mixed up with the light rail tracks.

I've just finished adding the MAX light rail Yellow Line in Portland.

I've decided that streets being mis-positioned are not the biggest problem with Tiger data. Rather, layers and intersections are. Now that I have read the Tiger documentation, I see that it does clearly state that there is no discrimination among overlapping layers (overpasses, underpasses) and that intersection points are put where the various paths cross. It is what it is.

Portland has several freeway interchanges with bridges, multiple layers of overpasses, train tracks, light rail tracks, and surface streets below. Unraveling the layers can be daunting. I started out being cautious, trying to preserve as many of the original nodes and ways as I could. Now I'm finding that it is quicker to go in and delete all of the intersecting nodes and ways in the middle and just lay down new ones as needed.

Of course, I still don't have anything to complain about. It does still mean that 99% of the mapping is already done, compared to the countries where folks are starting from zero.

It's easy to collect GPS track logs when driving around running errands. Since I'm in the US, I don't need to gather street names for most streets; I just need tracks to align the Tiger data.

The easy thing to do is turn on my GPS logger, set it on the car dashboard, and drive around. When I get home, pick it up off the dashboard and turn it off. Easy to do, but only one problem: I am paranoid, and I don't want every public tracklog showing my driveway. Yes, it is probably silly, but there it is.

One choice is to not turn on track logging until I am a block or two away from home, and then turn it off a block or two before getting home. It works, but it can be distracting while driving. I could also edit the GPX file with a text or xml editor, but that is repetitive and boring.

The best solution is GPSBabel, which is made just for this type of thing. It has a processing filter which can include or exclude points that lie within a boundary which you define.

I'm already using GPSBabel to translate my GPS tracks from the Coto Palm GPS software into GPX format, so I figured this would be an easy addition to the process. It took more work than I expected, because of an obscure detail in the processing. GPSBabel only wants to apply the filtering to defined waypoints, and not to tracks. Fortunately, GPSBabel itself can solve this problem, too. I told it to translate the tracks into waypoints, filter, then translate back to tracks. It added waypoint names for every track point, but I could easily delete those.

The end result is a shell script looking something like this:

F=$1

NewFileName=` echo $F | sed -e 's/\.pdb$//i' | tr ' ' '_' `.gpx

#Convert trackpoints to waypoints, for exclusion to work,
# then convert back

… See full entry

I've been gathering GPS tracks for downtown and southeast Beaverton. While I have not covered anywhere near every street, I have covered enough to see that the Yahoo aerial photos and USGS Urban Area aerial photos are aligned quite well with the GPS tracks. All the rest of the streets in the area can be interpolated from the photos.

I've been trying to find some easier ways to clean up the Tiger data for my area around Portland, Oregon. I've been corresponding with a few others also trying to do the same thing.

I'll skip to the punchline first: Better JOSM layers for aerial photos at

http://terraservice.net/ogcmap.ashx?version=1.1.1&request=GetMap&Layers=urbanarea&Styles=&SRS=EPSG:4326&format=image/jpeg&

and public street maps from the county tax office at

http://columbo.nrlssc.navy.mil/ogcwms/servlet/WMSServlet/Portland_OR_Metro_Maps.wms?VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG:4326&LAYERS=0:42&FORMAT=image/png

Here's the long version of the story:

For my county, I've found excellent position consistency among all data sources other than Tiger. My GPS readings match the Yahoo aerial photos, Google Earth, Washington County tax maps, Beaverton city maps, and the USGS urban area photo maps at terraservice.net. The Tiger data is broadly correct with pretty much all the right streets in the right relationships, but it is often shifted out of place by varying amounts.

So everybody agrees except Tiger. This is good, in that it gives me confidence in the other sources, meaning I don't have to travel every single street with my GPS since my tax dollars already did so. I figure that if I get a large rectangle by GPS which matches the aerial photos, then the area within the rectangle can be aligned with just the photo. I've been looking for the most efficient way to do that. Sometimes the Yahoo aerial photos are not as clear as I would like, such as being obscured by trees.

Although I like the simplicity of Potlatch, I found I really need the multiple-node selection and movement capability in JOSM for realigning the Tiger data. The key, then, has been to find better sources of public data which I can use as WMS layers in JOSM.

… See full entry

I bought a cheap GPS on EBay. I got a Palm Vx with a Magellan GPS companion attachment for ten dollars. By PDA standards the Palm Vx is completely obsolete, but it works fine as a GPS data logger.

The accuracy is pretty good. I have no way of measuring it explicitly, but it usually matches Yahoo aerial photos and Google Earth within a few feet. That is for open skies, either from my jacket pocket while bicycle riding or on the dashboard of the car. Looking at my tracks in Google Earth, it does correctly match the parking space in which I parked the car at the shopping center. Scary.

It didn't come with any software, so I'm using cotoGPS as the track logging software and converting it to GPX with gpsbabel. It appears that all the Palm-related GPS or map commercial software companies are no longer selling the products now.

I'm working on editing Beaverton and surrounding area. So far it is all cleanups of existing Tiger data, comparing it to aerial photos and existing GPS tracks.